But both slaves and foreign residents ("metics") transmitted their status by birth: you could be born in Athens, as could your father, and your father's father, and still have no route to citizenship, unless the city chose to grant it as a gift (a rare event). So even if you discount the foreign-born, you still have an elite ruling over a disenfranchised population.