I agree with your point, though even if there is consensus on a topic, we should still be able to have open discussion without censorship.
If you assume consensus is infallible, you'll be wrong sometimes. If dissenting voices are silenced, it's even worse: you'll be wrong, and not know that you're wrong.
In the cases where consensus is right, you're only adding fuel to the conspiracy theories when you censor opposing views.
In summary: freedom of speech - even wrong and unpopular speech - is essential to our continual pursuit of truth. To pretend that our world is somehow different today (as if misinformation or wrong ideas didn't exist before the internet) is just ignorance. How many times does history have to show us where the road of censorship leads?