It's a heckler's vote. Now, the conversation is about Yarvin and his politics. But if these 3-4 radical and deeply political agitators had not made this an issue to begin with, everyone attending the conference would have just known they were attending a presentation on Urbit (if they even chose to check out that session).
In other words, if this is established as a precedent, then a passionate enough fringe group can create the "destruction" they claim to be against to justify covert ideological policing. That this is occurring at a strictly technology event with a reputation for openness to innovative, experimental ideas is particularly chilling.
"Nice conference you got there, shame if this speaker I don't like created a controversy for you"-type manipulation does not seem as bad or obvious when the targeted beliefs are foreign to your own. Stepping back and anticipating the implications reveals a troubling future, and one that is occurring in many industries at once.