Downvotes change how you think about reading comments. Instead of "was this helpful?" you're thinking "do i love this or hate this?". It's polarizing, and as the number of users increases, it's easy for a comment to hit -4 in a couple minutes.
People feel bad when they get downvoted (often because they feel the downvotes are unjust), and some of them turn to trolling or flaming as a defense mechanism. This spreads across threads and infects the whole community. Emotions are contagious.
Also, add the ability to undo your upvote.