An HN pet peeve of mine is that flagging does not require even a modicum of rationale. I can flag any submission without even the slightest justification. When my own submissions have been flagged, I've also been left wondering "Why?" I understand why many of my submissions have not been upvoted. Many are simply not of upvote-quality. But not upvoting is quite different from flagging.
You may be interested in the HN Slapdown user script [1] which provides insight into articles' scores versus their rank order. With this installed, I tend to look especially at those with red scores because I am interested in counteracting what I believe to be an overzealous application of flagging.
Even right now, this very article has a yellow score indicating that it may be flagged (or something else is affecting its rank order versus its point score).
I personally have only ever flagged one or two articles, and only when I felt that the subject matter had utterly nothing to do with my understanding of the purpose of this site. When I did so, I was shocked at the magnitude of the penalty: if I recall correctly, the article I flagged was instantly dropped from the front page to the third page. It really gave me appreciation for how sparingly one should apply flags.
When I don't find a submission interesting, I just don't upvote it.