There are two problems with this approach.
First, there are more comments that are either factually wrong or interpersonally toxic than I have time to reply to. Downvoting is quick.
Second is the recursion problem. If I downvote and reply "factually wrong", then some people (either trolls or mistaken) may downvote my reply and say "no, you're wrong!" To say the least, that doesn't lead to a productive conversation.