Every citizen and non-citizen should care about the census, because the census and the data used and collected in the census, is used to apportion billions of dollars of resources. To address you data and privacy concerns, it's important to note Title 13 which protects or requires the federal government to protect all of the data that is collected.
If you suspect fraud, call 844-330-2020 to speak with a Census Bureau representative. If it is determined that the visitor who came to your door does not work for the Census Bureau, contact your local police department.