I first want to clarify that I'm not advocating for social Darwinism. I'm using it as a foil to show how our society is NOT darwinist, as we have well established institutions who's entire purpose is to prevent death as a result of fitness failure.
My point with the poor(referring to the welfare class, not the working poor)/young/uneducated being unproductive is to point out merely that they are. WHY they are is for a much larger discussion.
That said, I wouldn't be overly concerned about people being temporarily disenfranchised. As you mentioned, the overall trend is for it to expand regardless of obstacle. A trend not to be taken for granted, but one which can be comfortably taken as faith.
Population and demographics matter far more over time. They represent the upper and lower bounds of the groups that can be enfranchised at all.
In short, even if some elections are bent by some actors to make victory more likely for their side, if they aren't cementing that victory in the form of future generations of whatever they define as their ingroup (biological, cultural, creedal, etc.) then their gains are temporary and will erode with time as they are simply replaced and new group interests take over.
And yes, if humans engaged in the social equivalent of memory garbage collection that would look an awful lot like eugenics. The tricky thing is that eugenics, when applied to actually heritable traits as opposed to merely associated traits (think Crohn's disease or sickle cell anemia vs something like income), does 'work' in that it reduces the % composition of whatever negative factor is being targeted. People are upset about the word 'eugenics' because it was so closely associated with racism. That made it morally repugnant, but unlike, say, phrenology before it eugenics is far more 'correct' in that it's predictions are accurate over time. The real heart of the remaining valid controversy over it is what traits are targeted for expansion or removal and how/if that is enforced in practice.
For example, in almost all developed countries, pregnant women have samples taken to do genetic screening on their children early on in pregnancy. These can detect things like aneuploidy, NTDs, heritable conditions (like sickle cell), and even facial features if you check for them. While facial feature results are uncommon, it is very common for women to be recommend a termination if their child will have some major genetic condition. This is eugenics, yet we have found it more help than not.