What happens when x people downvote the same comment? You have a string of at least x replies that would likely be useless ("lame", "troll", etc.) Then someone would be inclined to downvote those comments, etc.
The majority of my karma has come from comments instead of submissions. One of my recent higher ranking comments ended up with 91 upvotes, but I know from watching the thread that probably 2 or 3 people downvoted the comment. In the end, why does it really matter? There are enough users here that comments and submissions will gravitate towards the correct ranking. Comments that are massively downvoted tend to have an obvious reason why, even if the reason is clear but not rational.
I don't think your solution solves any real problems, and inhibits peoples ability to moderate comments in a low-effort manner.
I have seen several cases where a comment is left like "I down voted your comment because..." In the other cases, forcing a comment would either add more noise to the conversation, or discourage people from downvoting and cause the comments to be falsely ranked.