I think that discussing both sides of a cost-benefit calculation will always sound like “downplaying” one side if you happen to think that one side of the calculation is obviously and straightforwardly the weightier. The way a utilitarian would approach this (not my preferred framing but not a bad one here) would be to say “how many deaths did vaccine misinformation cause, in expectation, probabilistically?” which is very hard to estimate, and then “how many deaths would result from the elimination of Enlightenment free speech norms qua norms?” which is even more impossible to even estimate. If we have a nuclear war because anti-war speech is suppressed (again: a thing that has happened historically) and three billion people die then “obviously” speech suppression was more harmful than vaccine misinformation. But this is the kind of thing that will only be obvious when it is too late.
To put it succinctly, you think I’m minimizing the harm of vaccine misinformation, and I suspect you’re minimizing the probable harms of normalizing speech suppression, and maybe we’re both somewhat right about the other.