Yes, this is the worst aspect of HN's design in my opinion: the paler font is essentially the equivalent of silencing dissent.

What is worse, it is an entirely silent way of silencing dissent. I would love to experiment with the other end of the spectrum: not only should the number of downvotes be visible, downvoting someone should also require a reason to be given, in the form of a post. And it should be possible to downvote those reasons too, with the algorithm adjusting the original downvote's weight based on the score of the downvote reason.