One reason that guideline exists is that unfair downvotes frequently get canceled by users who come along, see the situation, and make a corrective upvote. Meanwhile complaints like this linger on in the thread, inaccurate and off-topic—they don't garbage-collect themselves. As an example, I noticed your other comment and upvoted it before I saw this comment here. Similarly, other users have upvoted the GP.
As with any stochastic process, there is a lot of error and spillage with downvotes. There's no way to perfect it; you have to ask whether the system is better off with it than without it. Forcing comments wouldn't help, and posting complaints certainly doesn't help.