I have some pretty unpopular opinions because of my support of this Constitutional underpinning. For instance, I think that there is nothing wrong with exclusive country clubs. Or hoisting Confederate/Nazi flags on private property. Or even kneeling during the anthem. Etc.
Our government does not stop any such activities. (And if they did, I would consider it an act of governmental overreach on the level of treason against the Constitution.)
Now from the other side of it, if that exclusive country club does not want me on their grounds, that does not restrict my activities. I don't have any rights that impinge the rights of others. That is, the country club does not have to give up its right to be exclusive, because I want to go eat there.
My rights, shouldn't trump anyone else' rights, but I believe my rights should almost always trump the government's rights. And my honest belief is that that's how the Founders saw things as well.
That's the way I think things should work.