In US 43% pay no income tax
money.cnn.com
money.cnn.com
You wish to disenfranchise students who get a subsidized student loan and spend all of their time on their studies?
To disenfranchise farmers who get subsidizes for fallow land (no "crops of any type on this land, for each year of the last 5 years") where the farm isn't in operation?
To disenfranchise my disabled and retired father who gets subsidized support from the government?
How much work does one need to do to be worthy enough to vote while getting a subsidy of some sort? One hour? One day? Does it matter the type of job and the amount paid?
Should someone making $1 million per year have a higher quality and quantity of votes than you do. How many votes should Bill Gates have? If there is an upper bound, what is it, and why?
Regardless, this "%d don't pay income taxes" canard shows up every year, and it remains an incredibly misleading number.
First, those same people pay plenty of taxes -- payroll taxes, state and local income tax, property taxes and, of course, (highly regressive) sales and excise taxes. You pretty much have to be living in a self-sufficient monastery not to pay taxes of some sort. And those who don't pay income or payroll taxes are the unemployed, the disabled, the elderly, and students. It would be exceptionally strange to levy payroll taxes on people who aren't on a payroll.
Second, the past few decades have seen many social programs, favored by both sides of the aisle, provided by IRS rather than traditional program agencies or via block transfers. The Earned Income Tax Credit (families!), Child Tax Credit (children!), Head Start (babies!), educational tax credits (college!), mortgage deductions (homeowners!), charitable deductions (churches!), retirement plans (workers!), health coverage tax credits (high-deductible health plans!) and employer-subsidized health plans (insurance companies!) and other similar programs are administered as part of IRS' remit. (There are also some favorable tax treatments of certain farm program payments, depending on how the monies are spent.)
The majority of non-income-tax-paying households are not too poor to pay taxes (I'll note in passing that the President who took the very poor off the income tax rolls was Reagan), but middle to high-income households who gain more in credits than they pay in taxes. Tax expenditures are regressive -- representing a tiny fraction for the poor, and steadily increasing as a percentage of income through the top 1%. To be clear, the more disposable income you have, the more avenues you have to reduce your tax burden (even without moving money offshore).
It's also misleading to look at whether a person is paying income tax in a given year. The number of people who never pay income tax is incredibly small, because it consists of people who are desperately, chronically poor (and generally deeply rural) or very disabled. Eventually, pretty much everybody ends up paying into the kitty at some point.
Finally, on a personal note, I can say that my family, though we pay a lot in taxes (and probably more than we could, if we were more aggressive in our accounting), are lucky enough to never feel overtaxed. It's a bit depressing realizing that almost forty cents out of each dollar is going to the military and intelligence communities and over ten cents to debt service (I'm not counting SS and MC, since they have nailed-down funding sources), but I feel that, overall, funding expenditures that contribute to science, infrastructure, education, and the impoverished is a Good Thing, and I'm lucky that I'm in a position to contribute, rather than require assistance. Our success should make us happy, not make us angry that other people aren't as lucky!
There's also the 18 cents federal excise tax when one buys a gallon of gasoline, the $1.01 per pack of cigarettes, telephone tax, and other taxes.
(Personally, I think payroll taxes should be included as coming from the employee, and that it's mostly a bookkeeping detail about who nominally pays the tax.)
In addition, your simple threshold is meaningless. How long may I not pay taxes before I lose the right to vote? If I take a year off of work, make no income, and live on savings, then I'll pay no income tax. Should I lose the right to vote because of it?
If my wife goes back to school to get a Master's degree, and I support her for two years, the she'll pay no income tax because she doesn't have any. Does she lose the right to vote?
What of retired people, living off savings, or who have arranged their accounts so as to not pay federal tax? (See http://www.forbes.com/sites/baldwin/2013/06/05/how-retirees-... )
And so on, and so on, and so on. Are all of these people "drag[ging] the country down, quick"?
If not, how do you determine who is and who is not a drag?