What if you get called the fascist then? Anything you say to defend yourself is just fascist mind tricks. Denials are just more evidence of guilt. I would still defend your right to call for us to be silenced.
Also, nobody is being silenced here either because there is no obligation built into the first amendment or even the very concept of Free speech to provide a platform for anybody.