Okay but what about when I want to use the pulpit? By your logic, if you're using the pulpit you are restricting my speech, because I can't exercise my speech while you're exercising yours. And by that token, if everyone in the town square wanted to shout you down and drown you out while you were at the pulpit, you can't really complain on the basis of free speech, right? Because any restrictions on their shouting would encroach on their free speech rights.
Apparently this is what we call "cancel culture", and the reason we're still talking about it today is that people complaining about it have no coherent ideas on how to fix it without also trampling all over the very free speech principles they are decrying have been violated.
This knot people have tied themselves into is fascinating.