Regardless of whether or not you think a racist should or should not present (be paid to speak? Edit: they are not paid to speak at this conf although some expenses are covered) the founder of a pro slavery, pro facist movement speaking at your conference is a different sort of thing.
Unless he is speaking about pro slavery and pro fascism, I have absolutely no reason to care.
Can we please learn to separate technical concerns from nontechnical concerns? And before anyone starts replying with "they're one in the same!" - only under certain circumstances. If the subject of the talk is garbage collection in a programming language, what the speaker believes about anything other than garbage collection in a programming language is IRRELEVANT.
And then Karunamon comes in and asserts that no, they can't have freedom of association and they must judge others by Karunamon's standards only.