You should flag a comment when you not only think it is bad, but that the entire class of comments it belongs to should never be on HN. Flagging alerts moderators and, increasingly, algorithms. It's ok to flag as well as downvote a comment, since they do different things. But flagging should be rarer than downvoting.

As for exactly when to downvote: that's harder to put into words, but here are my half-baked thoughts.

Don't be indiscriminate. When a comment is blatantly false, downvoting is probably ok, especially when there's something else wrong with the comment. But high-quality, polite posts don't deserve penalizing just because you don't share their position. When you see one of those in negative territory, please be a good citizen and upvote it back to par. Users doing that is one of the community's self-correcting mechanisms, and it's more important now that we've made some downvotes more powerful.

Two things we can say for sure, though. One is that unsubstantive and/or uncivil comments can fairly be downvoted; the other is that complaining about being downvoted is off-topic.