I also happen to hate people with purple hair and decide to ban them from my property and refuse to service to them. Now this is my store and property that I own, these are my goods that I sell. Why should the government be able to legally control what I choose to do with my property and my goods, if I want to pour them into a sewer, or set them on fire should I not be allowed to do that? So why should the government force me to sell them to people I do not want to?
So the reasoning behind carving out every specific protection revolves around the idea that there should be a good reason for the government to control what I do with my private property, as it extends to reducing my liberty to do with my property what I want, and now becomes a matter of balancing various parts of the implicit social contract.
Whether or not this is justified or which side is right is a different discussion but that is why each exception needs to be carved out as I understand it.