I want to understand your reasoning better. If a policy is fair, but disproportionally affects a certain segment of the population - is it bad? Why?
The whole reasoning as to why exactly racism and ageism is bad, in my opinion, doesn't have to be based on morals, IMO (since everybody has different morals). It's sufficient to base it on self-interest: in an ideal simulation, where there would be bazillions of employers competing over bazillions of employees and vice versa, racist and ageist agents would lose, because they wouldn't make optimal decisions. (Of course, ideal simulation isn't real world, and that's why we have anti-monopoly and other laws that help to make the system run as if there were a bazillion rational agents).
But if some rational decision making merely correlates to segments of the population, and is not directly based on information such as race or age, I fail to see what is bad about it and why it should be avoided.