Unfortunately, there's no rulebook conference organizers (or their Greek choruses on message boards) can consult to say when rejecting a speaker is a good idea or not. It's a judgement call.
So when you've got a speaker with a reputation as a overt, 1950s-style racist in person, and a miles-long online track record of obscurantist racism, the kind of person who when accepted to talk at a technical conference about programming languages will write many paragraphs suggesting that science has established white people are smarter than black people (but maybe that shouldn't matter to us so much!) in his post announcing his talk, then yes, maybe you should avoid inviting that person.
On the other hand, you've got a speaker who used the word "promiscuous" as a pejorative in describing technology, and who spoke disrespectfully about someone else's favorite conference.
If you accept the irrepressibly racist speaker to your conference, you're going to lose most of your sponsors. But if you reject the person who implied promiscuity might be problematic in a metaphor, you're going to fail too, because anyone will be able to get any talk punted from that conference.
Maybe there's something else going on that we're not entitled to hear about. Those kinds of things also exist.
By the way: the implication that flexing and talking about "strength" is sexist is itself sexist, and not in the bullshit "it penalizes men and they have rights too" way, but in the "it reinforces stereotypes about women" way. There are women out there who can kick your ass.