This would not remotely "make the whole downvote system useless". The primary use of the voting system here is to sort out discussion by subjective value. That would only be very indirectly affected (through the very distantly secondary use of karma to confer downvoting ability).
If anything, it would debatably enhance the primary use of the downvote system, by giving authors of unpopular comments an incentive to remove from the discussion what the community has judged to be noise.
Though I would personally hate that, as a fundamentalist anti-deletionist, and as a frequent dissenter from this community's judgements of value.
Jeeez.
Edit: My upvote brought him back to zero, for now.
Rational minds can disagree, and intelligence can't understand much-greater-intelligence (so lessers might downvote much-greaters for that reason, besides the first-mentioned reason), but you don't have to worry about armageddons of downvotes. Or the balance of votes.
Just upvote what you consider "signal", and downvote the worst kinds of noise.
Thus you, my parent comment, are barely voted on at all, while the one by jewbacca rises to the top all day, no matter the weather.
In My Humble Opinion downvotes are given out a little too easily here (OK, I'm ready for more downvotes). I hope when I get the downvote ability I will use it wisely, not that many aren't wise.
Everyone wants peer approval. Also you get the same feeling you get when you are being denigrated by someone who doesn't have the backbone to come out and say who they are. I say you get the feeling, not that it is true.
P.S. I apologize for all the posts. I'll tone it down now.