Usually when I downvote its by accident, from me meaning t upvote but miss, really annoys me.

Only legit things I would downvote would be trolls.