The US used to use this logic until around 1960. It was a horrible country, at least if you were black. Even if a white person wanted to offer housing to black people, the white person was discriminated against. Collective private action used to reinforce a white supremacist social structure.
That's why we needed laws to require public services be available without discrimination based on race, religion, and various other protected classes.
"A private business can refuse anyone. That's because it is private."
A business may be privately owned while doing business with the public. That is quite different than a privately owned business which operates a private club.
Furthermore, businesses are subject to regulations that private people do not have. This is clear in organization structures like a limited company, where the company is a recognized as being legally distinct from its owners. These companies exist only because the government says so, which means the government can control what they can or cannot do.
It's also clear in places like licensing, where someone cannot simply say "I am a medical doctor" and offer medical services, and in other regulations like food safety, where restaurants -- even privately owned ones -- are held to standards which people don't have when cooking for themselves.