Voter rolls should only contain US citizens.
Census is "actual Enumeration” of “all persons", so you'd need to join/filter the data to remove people who are not citizens. Doing so would cause some folks to be afraid/unwilling to fill out the census.
2020 census is especially difficult because they haven't done as much (any?) door-to-door canvassing that usually happens during a census.