Students actually don't have a right to forcibly block speakers. That's a common misconception of free speech.
If you want to see examples of how real free speech is upheld, check out cases on thefire.org [1]. It's a law firm that specializes in tackling free speech issues on college campuses. However, they can't take a case without a plaintiff.
So, if one group protests and succeeds in blocking some speaker, the burden is still on the victimized group to file a lawsuit.
As they say, freedom isn't free, and you need to fight to keep it.