A lot of machine learning sounds like the computers are just cargo-culting. But this concrete example is spectacularly so.
The machine learning is "cargo-culting" in many instances because humans are cargo-culting and thus are creating models that repeat the same motions rather than try to provide qualitatively better data.
But the interesting thing is that in doing so it does reveal a lot of things about human decision making that may help inform improvements.
Imagine a misoginistic, but horny, boss, that only hires good working ugly men and hot dumb women, specifically looking for these traits in order to have a "playing" field. If the AI got in here it would be constrained to the same parameters, af if it was looking for good workers of any gender it would prefer men.
An the thing is that although this boss is part of history (we hope) the influence they had on the workforce at large was enormous, and it'll be a long time before this is reversed.
And this without taking into account all the coworkers that could hinder your job because you are not part of their cadre (gender, football club, fortnite clan, whatever).
We really should exepriment with equality, or even the opossite extreme, before considering one gender "superior" than another for a given role. And even then there would be exceptions.
Suppose men were providing $10 in additional value over the time they work for you. That's barely even worth a Starbucks gift card, the sort of thing you probably give away to people who guess the number of marbles in a jar at a company mandatory fun event. You definitely shouldn't hire James rather than Deborah because of an average $10 gender difference unless they are literally interchangeable. If their sole distinguishing feature was somehow that Deborah's smile was nicer, or James' laugh was more annoying, that's probably still enough difference to out-weigh this $10 lifetime value. And the reality is your candidates just aren't that similar. Deborah's six years experience in a similar role while James is applying with no relevant background should not lead to the conclusion that all that matters is the average "value once hired" for James will be higher based on gender averages.
Now, suppose women are providing $10M in additional value. Well, I think we'd have noticed something that dramatic by now right? "Gee, my start-up with five women broke even in the first six weeks because of how awesome women are". So we can rule that sort of large figure out as implausible.
I think a discussion about whether it's reasonable needs to take the magnitude of the difference into consideration, which would first mean (if you wanted to do this) finding out for sure how large that difference even is. My guess is, if you did this, you get a very small answer and so then the discussion is really easy: No.
But before you could do that, you'd need to have a decent objective measurement of this "value once hired", and that part is far harder so it won't get done.
You also need to consider the risks and circumstances, sometimes you want every tiny edge possible.
Analysis paralysis, figuring out what to do costs time and time is expensive so you should avoid this unless you've got a good reason to expect it actually matters and you believe you can actually do it correctly.
In the $10 example, I'd argue chances are very small that you actually know who provides the most value because just a tiny shift in the measurement criteria could shift things in the other direction.
If such a tiny edge matters, it's an indicator you have a problem.
But it would still be important to understand how the AI would assign value because it would matter to ensure that it does not apply criteria that are not legal.
At the same time, what this seems to suggest is also that purely judging by other factors in resumes might effectively act as a proxy for directly judging by gender. As such the more interesting question to me is whether that means that deciding based on such factors might put employers at risk of claims of gender discrimination.
For an AI to actually predict which employees are going to benefit the company, it need to understand both how the company has evolved over the data period, how it will likely continue to grow, and that some employees that have successful careers are not necessary the people that contribute to the success of the company. An AI might think nepotism is a great indicator for a highly retained and employable employee, but a human would recognize it for what it is but may accept it as an acceptable level of corruption. It would be the same conclusion (ie, hire the person), but for two very different reasons.
That's precisely what ML is. Though to be fair, interviewing done by real humans is also largely based on cargo culting.