Playing Devil's advocate, let's imagine I pour my life savings into opening up a fruit stand, it does well I move to a fruit store that is popular and everyone loves, at this point everything has been bought with my personal funds, it is basically my property, I run it as a sole proprietorship instead of an LLC, I hire people, but ultimately everything is bought paid for and owned by me.
I also happen to hate people with purple hair and decide to ban them from my property and refuse to service to them. Now this is my store and property that I own, these are my goods that I sell. Why should the government be able to legally control what I choose to do with my property and my goods, if I want to pour them into a sewer, or set them on fire should I not be allowed to do that? So why should the government force me to sell them to people I do not want to?
So the reasoning behind carving out every specific protection revolves around the idea that there should be a good reason for the government to control what I do with my private property, as it extends to reducing my liberty to do with my property what I want, and now becomes a matter of balancing various parts of the implicit social contract.
Whether or not this is justified or which side is right is a different discussion but that is why each exception needs to be carved out as I understand it.