You gain the privilege after you've accumulated enough 'up' votes for comments and submissions. The exact number changes over time due to inflation.
Interesting; I didn't know that. Right now it's at about 200.
Exactly at 200 - Once you have 201 karma points, downvoting functionality is unlocked.
OT, but it's always amusing to see an answer to this question (which comes up at least once a month) hit double-digit points. The rich get richer...
Until you have a certain amount of karma, you cannot down-vote. When you have enough karma, you'll see down-vote arrows below the up-vote arrows.
How do you downvote below -4?
Click the down button: it'll knock them down another karma point and fade out their comment more, but it won't display lower than -4 to prevent pile-on downvoting to -50.