Taking it further, what if a majority of businesses gradually decide to be racist and refuse all services just because they can? Not serving minorities wouldn't really impact their bottom line all that much. The minorities would literally die off.
It's easy to talk about "rights" as if they exist in a vacuum i.e. my rights are mine and they do not affect anyone else, ergo my rights should be absolute. They are not, and should not.
Reality is usually a tenuous balance of rights (usually tilted towards the majority) that people participating in civil society share.