These people are surprisingly easy to spot, by the way. They often come from a non-academic background, have a party or "think-tank" background with some obvious political agenda, and their funding also comes from corresponding organizations. Some of them are even just politicians.
The problem with these kind of speakers is not that they speak at a university. The audience is smart enough not to fall for bullshit and to separate politics from science. The problem is that these people use events to give their name and their agenda some credibility and make it easier for the public to confuse their opinions with science. It's a branding problem, political talks can give a university a seriously bad reputation.
Since it would be inconvenient and undemocratic to ban all political speeches and events from universities, I think a trade-off has to be made and some cut-off points are necessary, even if that means that a few interesting talks get lost due to being identified as false positives. It's perfectly fine and reasonable to ban all political speakers from the political fringes (both far right and far left) unless an event clearly serves a scientific purpose and the speaker also has the fitting credentials in terms of actual publications.
I understand this goal is hard to reach in the US, though, since many universities there are private and lobbyism has extended its reach to them. But it's an ideal to strive for. The point is not to keep politics out of universities, there is no reason for that, but to keep bad faith actors out of them.