It's a difficult situation to be in.
More reasonably but still on the aggressive side, does this mean that now we have to publicly announce when speaker invitations are rescinded to get ahead of the story?
I don't really think there's a way to win this. In the event that you're wrong, it'll blow up. In the event that you're right, it still probably will blow up, depending on whether that person (or their fans) go on the offensive or not.
The only actual winning move is not to have played (only inviting people that you know, doing advanced due diligence on anyone a degree of separation away), but are you really winning if you do that? It would seem to seriously limit your horizons.
I think it is wise to publicly announce both speakers who have accepted invitations and speakers whose invitations were revoked (don't even have to say they are revoked, simply stating "unfortunately John Doe will not be speaking in our conf" should be sufficient). If not for these situations, then simply to inform visitors whom they can expect to hear in the conference.