We can reasonably be very concerned about excluding a genuine contribution from a scientific conference, because that would be a case in which our personal politics were directly impacting the cite record, which in turn makes it harder for everyone to "do science". There are people in my field that I like more and people I like much, much less, but I didn't have any difficulty putting that aside on the PC of the last conference I was involved in: you rate the paper, not the author, because you have to.
That's not the case at an industry event, because scientific progress doesn't depend in any way on events like Lambdaconf.
To your point about the attendees: sure! If attendees of Lambdaconf really want to see a talk about a marginal distributed system by Yarvin, it is legitimate for the conference organizers to cater to the market by having him on. Similarly, it is legitimate for others to publicly express market preferences for avoiding conferences that feature him.
Between your comment upthread and my response, I think you're left with the trickier argument to support.