To me, the sets of "downvote-worthy" and "response-worthy" comments overlap very little. I downvote when a comment is not even wrong. When I comment in response to another comment, it's because I've seen some sign that the other person might be interesting, correct, amusing, curious, or any combination of those. A requirement to compose "justification" comments would just about end my downvoting.