The distribution of voting power is based on population, not citizenship. Non-citizens need to be counted too.
Slaves couldn't vote either, but in the famous compromise were counted as a portion of a person - the distribution of political power is why.
Women only got the franchise 100 years ago, but had always been counted on the census.