I've seen many perfectly-civilized comments being downvoted, presumably, because they are controversial, though I have seen others that deserve downvotes because of their nature.
The way I see it, replacing downvotes with a comment flagging system would probably reduce the amount rude comments because they'd probably be killed soon enough. However, it eliminates the penalty of being rude (a drop in karma), and some visible way of penalization must be put in place for it to be effective in deterring users from being rude. A work around might be that for every killed comment (due to numerous flags) a user gets -5 or -10 karma.
Things would behave differently with posts, but these are just my first thoughts (when it comes to commenting).