That's an abuse of the flag button. You flag a story that is off-topic for the site, or that is spam, or that deliberately trolls the site. But flags are not downvotes. I can safely count on the fact that dozens of stories I won't like will hit this site every day. I don't flag them.

You should hit the "unflag" button now. Or don't, but then don't complain when, after gratifying this bad habit of yours for a few more months, you lose the flag button.