Allowing people to discriminate based on cultural markers like baggy jeans instead of skin color is creating a loophole. It's analogous to the voter disenfranchisement laws after the civil war. You can say that you're not barring black people from voting, that you're only barring people who fail a literacy test. But if the effect is that you mostly ban black people, then it's a de facto racial discrimination.