This is an interesting line of thought, which I don't know if I agree with or not. It might be worth exploring the contrapositive: "If a piece of speech is tolerable to at least someone, then it shouldn't be censored."

And that raises an absurd but still interesting thought experiment: if by some process, someone is able to create a statement that is intolerable to every single person on Earth, would it be okay to censor it?