I disagree. It's very useful to remove the comments of spammers, trolls, and the like. Another common reason to downvote is to stop the tide of noisy comments, such as meme posts, which feel increasingly prevalent on HN.

A more exemplary argument might be to imagine hypothetical responses to your post:

"I agree."

"I disagee."

"Fuck you."

"lol"

"¯\_(ツ)_/¯"

I think you'd want to downvote something like that. Nobody wants to waste their time reading it.

I think the number of times I've missed a valuable comment due to downvoting is probably far less than the number of times I've seen garbage posts. We might disagree on the value and merits of each approach, but I'd prefer a high signal HN, even if it makes it a bit more of an echo chamber (hypothetically).