Is there not the right of freedom of association, and by extension, not to associate with those who offend without purpose?
Of course there is. You are free to not acciciate with those who offend you by leaving. You do not however, have the right to stop people from offending you by assiciating with them.
(Or maybe it doesn't. I dunno, really.)