Perhaps the part that you are not seeing is that if the poster truly cares about engaging with the community he or she cares about understanding why their post is being down-voted. A lot of topics usher-in fanboi downvoting which is petty and unfair. A down-vote with an explanation is good because it will keep the down-voter from impulsive action and create a better relationship overall. It is possible that the down-vote isn't appropriate, which could come out of a discussion stemming from the down-vote comment.

HN could easily make down-voting comments a requirement and then add a simply feature by which down-vote comments are folded below the original post and not visible. You could then have a "view down-vote comments" link that would reveal that thread. In other words, keep them out of the main conversation so that they don't clutter things yet make them available for review by interested parties.