Is there any reason whatsoever to NOT count slaves and babies as persons? You can not count them as citizens or adults, but surely they are persons.
For example, the original Roman census, from which we take word 'census', registered and counted adult male citizens. They did list their wife, children and slaves as part of extra information about their household, but those dependents weren't part of the census itself, just an "attachment" to the recorded citizen just as the notes about their land and other porperty. Other people like single women or orphans would get reported by their guardians, but they as well explicitly weren't included in the "count of heads", which counts only adult male citizens i.e. those who can run households and own property, be recruited to fight, pay taxes, and vote.
In effect, the ancient world doesn't really consider persons or individuals, they care about households (represented by a single individual) and so register and count those.
Remember that not that long ago at this scale, the USA considered slaves as 3/5 of a person when determining states' populations for representation and taxation purposes.