My problem with downvotes isn't the effect on my score. It's the fact that the font becomes paler. Dissenting opinions are singled out in a way that makes them look bad/wrong. I also don't like how the UX doesn't represent the distribution. A post with no votes will look like a post with 50 upvotes and 50 downvotes.
I am fine with downvotes, and with some UX mechanism to let people know that a post is being downvoted. But I think the current UX engenders groupthink.