The downside is that some qualified candidates (white man and woman of color alike) will potentially miss out on jobs once the candidate pool is grown, because you've pulled a bunch of extremely talented people into the pool who set the bar higher. That could certainly make people feel discriminated against if they don't have enough insight into the process.