People should be downvoting because its not appropriate intellectually (adds nothing, eg provides no evidence to back up a claim that obviously needs it). Flagging should be because it is simply inappropriate for usually offensive reasons.
I rarely comment now. I often don't see the point.
I've also often considered a tool to only provide the RSS feed for interesting article discovery then mostly ignore the comments completely. Comments on the ask/show sections are usually more signal than noise. Sometimes even the noise is interesting.
Upvotes should exist to make interesting content rise. Uninteresting content then simply sinks to the bottom.
But votes are a difficult problem to solve. I'm sure there are too many corner cases I've no idea about.