The biggest problem is that HN would turn into a discussion about downvoting, which is exactly what we are trying to avoid.
There are a plethora of bad comments on this site. Downvoting them so they fall to the bottom, without commentary, is how the higher quality discussion stays on top. Adding in commentary would turn the whole site into a meta-discussion about content moderation, which is not the goal of HN.
I could see an alternate solution to the problem, though - a private poll of why the comment was downvoted, visible only to the person who wrote the comment, so they can better understand why their comment was not desired by the community.