It would be completely out of character for him to pull out of the speech - far better for him to go along and mock them. I for one would love to see that.
It's like the reverse incidents where students at some universities demand controversial speakers not be allowed to speak at their university as if censoring a speech is a victory.
Let people speak and if what they say is wrong then debate it to show what's wrong don't hide it or hide from it.
I wish to give a talk about how space aliens built the Brooklyn Bridge using telekinetic herring, and give the talk in your kitchen. If you neither grant permission nor tell me where you live then obviously you have something to hide and wish to censure my speech.
I think herring is a fish.