Except we can, and we did, and society was better for it. So as it turns out, the "right to exclude" is more of a guideline than a rule, riddled with dozens of exceptions. And "the current regime has existed for a long time" is not very predictive of long-term societal shifts.
So it is here. You are right that there is fundamental tension between the right of free speech and the right of exclusion.
But one of those is a right so fundamental that it appears first when you ask people to list freedoms. The other is something we regularly cede in large and small ways when we live with other humans in a society. Where the two oppose, there is only one way the arc of history can go.
Our grandchildren will be just as puzzled by the "right to exclude" argument as we are when it is made by the generations before us.