You get a downvote button, separate from "flag", once you reach a certain karma level (I forget the number, though)

I think you need to be at a certain level of karma to 'earn' the right to downvote.

Flag is for inappropriate content / behaviour.

Karma threshold for the downvote button used to be 500, IIRC. Only 449 upvotes to go!