It doesn't matter - when the document was written America had tons of non-citizens and those people were counted.

Slaves couldn't vote either, but in the famous compromise were counted as a portion of a person - the distribution of political power is why.

Citizens of a certain age, who meet certain residency and bureaucratic requirements and haven't lost the right due to criminal convictions.

Women only got the franchise 100 years ago, but had always been counted on the census.