At a certain level (500 I think) you can downvote comments. No one can downvote submissions though. There is some evidence that your score affects how quickly your comments "sink" on a discussion page, but they never disappear or anything.