I am very circumspect about downvoting, and rarely do it as I am still unsure of the criteria. I rarely have the opportunity to downvote vitriol and trolls, as others have usually done that and there seems little point in piling on.
I personally have a much higher threshold for comments which are worded aggressively, sarcastically, or humorously. I occasionally find myself upvoting comments which have been downvoted by the community due to "tone," if I think the core statement has a bunch of validity.
Where I am most tempted to downvote is statements that repeat sticky memes that have been repeatedly disproven, but "sound true" within areas I have the most expertise, especially when they are presented as fact. Those kinds of comments lower the bar more than any other, IMO, by forcing a debate over whether the sky is in fact blue every single time the community attempts to discuss weather patterns and cloud formation. However, it is unclear to me whether that is valid criteria for downvoting, so I rarely even do that.
TL;DR I am unsure when to downvote so I mostly don't.