I came to the same conclusion. I downvote only if the comment is offensive. If it is wrong or I disagree (and I care) I add my own comment. Downvoting by itself adds nothing to the discussion. I think the current system works remarkably well and I dont feel the need for changes. I fear clutter more.