Oh wow. this happened to me recently. I foolishly believed that it was true since I've heard of companies like Reddit that have policies like these.
It isn't hard to write a comment about it that will show up in a search engine, and while that is far from perfect, it will allow employees who have freedom of motion to steer clear of slimy firms and those who don't to at least be able to negotiate from a slightly less-weak position.
One commonly mentioned alternative explanation is simply that people will quite reasonably not risk negotiating aggressively when they perceive their negotiating power as being lower. Given the number of women or non-Caucasian/Asian men who've reported being paid less or held to a higher standard than peers, I would seriously consider the possibility that the data is showing us a symptom rather than the cause of the pay-gap.
One of the problems is that when women lack such knowledge, they come across as less confident because, in the statistical sense of the word, they literally are less confident. In other words, in an absence of hard information, you cannot be sure if the figure you are asking for is too high, too low or just right. But when they evince this lack of confidence, people usually try to give them pep talks rather than hard data. It gets treated like a lack of self esteem rather than a lack of information.
I do think you are correct that women simply cannot afford to negotiate as aggressively and this is a factor. But I also think they lack access to knowledge that a lot of men are exposed to casually simply because they are male and this makes it easier for them to hang with the right crowd and get anecdotes, etc. And that piece is what I was trying to capture with the word savvy.
I am a woman. I know a lot about negotiating. I remain poorly equipped to negotiate on salary because I lack information. Having general savvy about negotiating technique is insufficient. You cannot play hard ball if you do not have hard information particular to the problem space in question.
1) I don't need a job. You'll pay me what I want or no deal. In those cases things like what this recruiter pulled won't work.
2) I need a job. In this case it doesn't really matter to me.
But they also can't keep bringing in people above a company's ask. It's a balance that has to be managed.
A lot of large companies have tiered salary levels that are not in any way tailored for the individual. In Germany, we have a Tarifvertrag in virtually every large company, which fixes the salary.