What bothers me personally the most is the way in which downvotes seem to happen. Don't get me wrong, a lot of users downvote in a good way, so it's not all bad.

Still, I often find grey comments that are brought in a completely respectable way. I feel like these comments are getting downvoted because someone disagrees with them, rather than because they are bad or toxic comments.

The opposite is also true, I often find comments with a positive amount of votes that are in the "internet jokester" style and don't really contribute anything meaningful to the topic at hand.