Algorithmic Monocultures in Hiring
hai.stanford.edu
hai.stanford.edu
I then went on to work for multiple firms that placed a premium on candidates from Ivy League/Top Tier (Stanford/Duke etc) candidates.
This taught me that:
- Their are pros and cons to any selection criteria.
- There are smart people everywhere. One of the smartest people I ever worked for spent several years in prison for drug dealing. He was on par with many of the Managing Directors I've worked for
- There was a study where they asked big bank recruiters which school consistently produced people who were excellent employees 2-3 years out from hiring and the answer was Penn State (not my alma mater)
- There used to be "manager's choice" hires where managers had 1 slot in a training program where they could select whoever they wanted. Sometimes that was terrible. Sometimes that person was top of their training program.
- Smart people are just as capable as creating problems as less intelligent people. Smart people, in some ways, are better at creating problems. Especially if the incentives reward them for creating those problems.
This seems like a great idea to me if you institute a feedback loop so managers who pick trash eventually lost the option to pick, and managers who pick rock stars eventually get more picks / more responsibility.
If we move to using just a small number of AI models to help do things like hiring, we will amplify biases and possibly completely lock out portions of the population. We need to be very careful when using AI systems to evaluate people in general -- not because they might be biased (which they might be), but because even a small bias, if used by virtually everyone, can be damning.
However, "this AI model can decide that some subset of people, perhaps random, perhaps not, are simply not hirable for any job" makes sense to most people regardless of political bent.
For example, the article here shows disparate impact: that different percentages of applications are passed through the AI filter. But it doesn’t show differential treatment of otherwise identical applications based on race.
Note that the OP uses "systemic rejection", while the paper does reference bias, it is in the precise meaning of the word.[0] And this is not targeted at the general public.
You may want to look into the 1990 GOPAC handout "Language: A Key Mechanism of Control" to understand why some groups would simply just weaponize any term that was substituted. Academic papers need to error on being precise, to be effective, not focused on handling the general public with kids gloves IMHO.
Edited to add, listen to Lee Atwater's 1981 Interview on the Southern Strategy for even more context.
I don't think this even requires any bias.
Assume there's some loose ordering of who is or isn't a good hire, and every employer has their own fuzzy view of it. If you get slightly better or worse as a potential hire (pick up an extra degree, let your latest certification lapse, whatever), it gets somewhat easier or harder to get hired.
Now assume that same ordering, but all employers share the same view of it. I'd expect the divide between employable and not employable to be much sharper.
Also, we will of course have all kinds of attempts to "game" the system to get ahead. Optimizing (even more) for the metric. Degree mills, for instance.
You might be able to avoid it with a panel of AI, similar to how we try to avoid it by using panels of humans, but even that turns out to be contentious and not surefire.
I have feeling with AI it'll be even worse, since folks / companies can pass the buck (similar to how health insurance companies are now using it to deny folks).
Unless you're taking the "there are multiple mathematically incompatible ways to define bias" view of the topic, just do what's already known best practice for high-bureaucracy human review. Which is too define an overly-pedantic standard rubric.
But LLMs are statistical models. They are aggregating all biases into a general super bias. And they're all converging towards the same solutions.
A lot of the capitalists see that as a positive.
Definitely open to opposing or critical views
The dataset is constructed, deliberately, to hold candidate performance constant and vary the names of candidates to appear to be associated with a specific race.
But they picked 9 family names per group. Which sounds quite low. And combined that with first names to reach 500 first+last names per group.
I wonder how much of the bias we see has to do with the names actually picked versus it being racially motivated (absolutely not denying that this probably is a factor, but might not be the only one).
For example, in France there is the national BAC end of high school exam. If you you at the names X grade distribution, and look at the higher “very good” bracket: some names are heavily under-represented (less than 5% of say “Jordan” get that grade) while some are over-represented (35% of “Josephine” get such a grade). The exam is for the most part anonymous, but some names are definitely heavily correlated with lower/higher income groups. So nothing surprising: Josephines tend to come from richer families, thus in average get better education/support, thus better grades. Same thing is true with family names to a smaller extent.
So I wonder how much of the bias we see, be it from real persons or the AI has more to do with a class thing than a racial thing. Again those are not neatly separate things, but still
Cool.
In any event, I'd happily support a ban on all parts of the ATS that could be involved in automated approval, rejection, or scoring being able to see candidate names. But I sense the author of this has a bigger agenda.
If you click through, the paper says the race is self-reported.
“Our data tracks 4,197,168 applications. It includes applicant gameplay features and for each application, the application date, the position name and employer, metadata about the position and employer, and the numerical score and final recommendation each applicant received for each completed application. 40.2% of applicants self-report race with a breakdown of 16.8% Asian, 14.2% White, 3.6% Black, 3.0% Hispanic, and all other racial categories below 2% (i.e. fewer than 100,000 applicants).”
its going to be in the rest of the data because race has a meaningful correlation, and pleanty of causation with being disadvantaged in real ways, that can also affect the ability to then do certain jobs.
like, the environmental pollution and building interstates and freeways through black communities, on purpose to do bad things to those communities, then results in a bunch of noise and particulate pollution, that is bad for developing brains.
you wont be able to do some meritocratic non-racist hiring without fixing the environmental racism. otherwise youre just mirroring racism other people built for you
I'm not saying AI is not biased, but this study does not prove that.
[0] https://arxiv.org/pdf/2605.27371
From the paper:
> Fig. 1. The pymetrics process. > Stage 1: Applicants apply to positions. > Stage 2: Applicants are directed to the pymetrics platform to play assessment games. > Stage 3: pymetrics algorithms use applicant gameplay features to recommend 58.2% of applicants per position on average. > Stage 4: Employers decide which applicants to interview or hire, typically rejecting applicants that were not recommended by pymetrics.
High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations
That's a pretty common sense legislation to me.
Of all the things listed "recruitment" doesn't belong to me. Is the argument that it is someone's fundamental human right to get someone else to pay them to do a job? Or is it strictly about human oversight?
There's no reason to single out AI vs any other approach to the same topics.
We find applicants are more likely to be rejected from every position they apply to than would be predicted by the baseline of each position making statistically independent decisions.
Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer. Like online dating, most employers are looking for some baseline indicators that you are going to be successful and stable.
Actually the fact that they found this result didn’t hold in a different dataset is especially weird.
This makes sense to me, albeit intuitively and in a way I can't articulate.
> an accepted resume is more likely to be accepted by every other employer
but this doesn't necessarily follow from the prior for me. Plenty of people get really good jobs and are really successful in them only after dozens or hundreds of rejections with a nearly-identical resume.
I just think certain resumes will get an interview almost every time in some industries and certain resumes will likely never get an interview almost every time, but the majority of resumes are like you say have different aspects that appeal to one empoyer over another.
But that wasn't the case for non-algorithmic screening. From the paper:
"By contrast, we find that when first round screening is not mediated by a single screening procedure, systemic rejections are close to the baseline. To support the empirical validity of our baseline, we study homogeneous outcomes in the largest study of first-round screening at U.S. employers to date. Kline et al. [38] generated 83000 synthetic resumes and submitted these resumes to vacant positions at 108 US companies between October 2019 and April 2021, a similar time period to our data. The companies, which are a subset of the Fortune 500,15 collectively employ 15 million workers. We analyze the homogeneity observed in the resulting callback outcomes in their data. We find that the baseline is an effective estimator of the systemic rejection rate for this dataset. As shown in Figure 3, the observed systemic rejection rate is accurately predicted by the baseline and a chi-squared goodness-of-fit test cannot reject equality of the two distributions (2 = 20.05, = 0.69). In other words, while the largest previous study observes systemic rejection rates consistent with employers making statistically independent decisions, the algorithmic hiring data shows significantly correlated outcomes that lead to higher-than-baseline systemic rejection rates."
Obviously as a hiring manager you're looking for a hard working individual with a number of successfully completed projects and glowing referrals from multiple places of employment, but you're also looking for a person with expertise in particular technologies/industries/whatever other areas of expertise. To a large extent requirements for each role are unique, however many do have some overlap.
So being rejected from one position might simply mean there's a misalignment between what the company is looking for and what the individual has. Which might not be the case with other companies.
So if we're seeing increasing number of candidates being consistently rejected at multiple places the question "why" is a valid one.
That seems like a nonsensical way to measure racial discrimination. What could justify it?
it sounds like how you'd get that kind of metric at least
Here's some analysis of what it is and why it's useful as a canary in the coal mine: https://www.prevuehr.com/resources/insights/adverse-impact-a...
> Since the 80% test does not involve probability distributions to determine whether the disparity is a “beyond chance” occurrence, it is usually not regarded as a definitive test for adverse impact. Instead, other statistically significance tests, such as the standard deviation analysis, may be used for this purpose.
But then my question recurs: isn’t this a ridiculous way to measure discrimination? It’s assuming that the only thing that differs between the different ethnic applicant pools is their ethnicity, which is essentially never going to be true.
Like. If I am evaluating a developer on lines of code written, I am a bad manager. But if an engineer has 40% fewer lines of code than the team median, it's absolutely ok for me to go, "Interesting. What's the story there? Are they slower or is there some other factor?"
Same idea -- this is purely a fast, first pass metric that can quickly assess if something warrants a deeper evaluation.
I expect Median LoC might be very high with the average developer using AI these days... but the dev who is making atomic changes that are fixing the AI output is probably tiny LoC but way more important
If you are trying to say "more data needed, headline misleading" you should say that instead of misrepresenting the 4/5ths rule. Also the word "can" implies uncertainty of conclusion. This isn't ridiculous, the authors point out that this is the first large scale study of this topic. Nothing has been "proven" here, it's showing that this warrants further investigation and attention.
Do you read many academic papers, because you seem to be having a rough go here.
The authors of this study have done that here. To borrow from your example, if you saw a statistically significant amount of my posts highlighting the merits of the Iranian government in a ways that run counter to the general global consensus of their actions, you would then have something that other people might agree was worth looking into. A hypothesis is not “misinformation”.
This article has not claimed to have proven anything other than outcomes in a process. I don’t understand why this is so upsetting to you.
The assumption that applicants from all races are on average equally qualified for every position. Whole subfields of modern academia are based on that assumption.
I am getting downvoted because it's hard to admit that AI only reinforces the culture that it is trained on. It is the perfect technology to keep systemic racism in place, all while being the perfect scapegoat for lack of personal or corporate accountability.
Individuals are qualified or unqualified. If a company happens to end up with less than 1/4 Ravenclaws or not very many Virgos, it doesn't mean hate is a reason. It could be that the Ravenclaws that applied were a bit less qualified than those from the other houses.
I guess my point is, doing the statistical analysis for race and gender and drawing conclusions, while being completely blind to the one single factor any sane hiring manager should be focusing on -- actual qualifications for the role -- doesn't make any sense.
Don't claim AI is discriminating against non–selects, though.
I doubt companies are using Gr*k to make their hiring decisions.
It indicates there may be adverse impact to one group. It specifically is not used to resolve racial discrimination.
It's purely a signal for "we should consider asking more questions, because this appears unusual". That's what your quote says too, it "flags" a low recommendation -- it's indicating further study and investigation is likely warranted.
"Adverse impact occurs when there is (i) practically and (ii) statistically significant disparities in the selection rate for the group of interest when compared against the selection rate ′ of the most selected group ′ . Practical significance requires the impact ratio ... to be less than 0.8, which is why the EEOC guidance is colloquially referred to as the 'four-fifths' rule."
The headline numbers reflect the positions for which the 4/5 rule was triggered, not the result of some further investigation: “We discovered that 26% of Black applicants and 15% of Asian applicants applied to positions where the AI system discriminated against their racial group.” Based on the methodology, I think that means that 26% of black applicants applied to positions that were flagged under the 4/5ths rule.
This doctrine is the basis for much of employment law. It is a significant reason why employers don't administer IQ tests (or equivalents) to screen candidates since ~the 90s.
A common objection to the doctrine is that it leads to unfalsifiable discrimination claims, which is why it seems nonsensical to you.
There is a large body of literature concerning the question "does disparate-impact enforcement cause employers to alter hiring behavior in ways unrelated to actual productivity or discrimination?" and the answer is largely "yes". As you suggested elsewhere in this discussion, Google may be useful.
Are you suggesting that companies should violate the law here? What do you recommend?
Edit: charitably, "adhering to the letter of the law" is sometimes shortened to "law-abiding" and is generally what we want.
Prior to the beginning of your excerpt is the word "You", meaning the comment's author is the subject, not "companies". I'm saying the commenter is appealing to black letter law for the answer to the question "what happens when..." but we have observational evidence to answer the question.
Isn't the point that the observational evidence amounts to the companies in question steer clear of illegal behavior?
There are anti-money laundering laws, so banks institute procedures to help them comply. Yes, we expect companies to change their processes so they don't break the law. That's the point of the law.
I am confused with what you think companies should do in this situation. Expose themselves to legal and civil liability? Or change their behaviors so that close scrutiny indicates they are trying to comply with the laws and any bad actors acted against internal procedure?
You're arguing about something else, using the form "given we know that racism is happening, and furthermore we know where and how it is happening, why shouldn't we just do these Totally Cost Free and Obviously Good Things that are just like fighting money laundering"
Well, I just don't think any of these things are as evident as you seem to assume they are. Also fwiw I grew up in the US, where I was called all sorts of slurs -- like, the real ones you probably censor in your internal monologue when you see them written -- throughout K-12 education. I still don't believe in the existence of pervasive and oppressive racism the way you seem to assume it.
To act like it's bad that people of colour have a more fair chance of getting employed because of some piece of legislation is simply insidious. It's just been over a month since black people lost the right to a fair vote.
Literally the opposite happened. The Supreme Court ruled that there was VRA §2 liability when there was evidence of racially-motivated gerrymandering: "In short, §2 imposes liability only when the evidence supports a strong inference that the State intentionally drew its districts to afford minority voters less opportunity because of their race." (Louisiana v. Callais, p. 26)
If the issue happens upstream of the defendant to a claim - generally an organization being sued by an individual with fewer resources - it incentivizes such entities to push for changes upstream, so that they don't get stuck with the bill.
If you're making the claim you need to provide the evidence.
Most people would say that a persistent disparity means it's possible there is discrimination, but it's not definitive proof.
We have a "disparate impact" and nobody can prove what proportion of it is due to things like parental income or childhood education as opposed to racism on the part of the employer. Because the former considerations are real contributors, the metric can regularly be expected to exceed the threshold even if the contribution of racism by the employer was zero. Doesn't that imply that we're essentially accusing people of racism at random?
> because the impact exists whether intent can be shown or not, the desire remains to ameliorate that impact.
The median household income for Asian Americans of Indian ethnicity is more than double those of Burmese ethnicity:
https://en.wikipedia.org/wiki/List_of_ethnic_groups_in_the_U...
This is objectively a disparate impact and likely shows up in several other metrics in addition to income. Disparate results can almost universally be obtained by arbitrarily segmenting the population into different groups and comparing the midpoints. Americans of Australian ancestry have a higher median income than those of Irish ancestry, Bolivians higher than Cubans. The result is often because the lower down group has a history of being oppressed.
What reasoned means can we use to determine which groups get the benefit of these methods to ameliorate the disparity and which don't? What should be done about the inherent impossibility of doing them simultaneously, e.g. because hiring a South African woman over a Haitian man would reduce the disparity on one axis while increasing it on another? Notice that considering each group separately could result in unconditional liability because either available alternative puts you over the threshold for one group or the other.
> If the issue happens upstream of the defendant to a claim - generally an organization being sued by an individual with fewer resources - it incentivizes such entities to push for changes upstream, so that they don't get stuck with the bill.
Do we want to apply this logic to other things? The median income in California and New York are significantly higher than they are in Alabama or West Virginia and they have higher ranked public schools. We can correspondingly expect that when applicants from different states apply for the same job, the ones from California and New York (even if they're the same race etc.) are more likely to be selected because they had more advantages growing up, even though none of them chose where they were born.
By the same reasoning we should then have the federal government penalize employers for hiring the applicants from the more affluent states so that it "incentivizes such entities to push for changes upstream, so that they don't get stuck with the bill." Does it make sense to do that?
That's the beauty of aggregate statistics. You have some elite job with 100 slots. Before you had 60 of the slots going to cronies and the remainder being allocated on merit. The cronies were disproportionately of the same ethnicity so your statistics are skewed, but don't worry, all the cronies still get their slots. Because statistics can be balanced by getting even more cronies, this time of a different ethnicity, and giving them as many of what used to be the merit slots as you need to manipulate the average. Using statistics is perfect for pretending that you're giving people something when you're actually taking something away.
Because surely no one would have legitimate preferences based on their gender, cultural norms, etc. or real differences in aptitude due to childhood exposure, education, or said norms and preferences.
They find "disparate impact" of pymetrics across racial groups, but it doesn't seem like they controlled for anything.
"Cards held by African-American sellers sold for approximately 20% ($0.90) less than cards held by Caucasian sellers, and the race effect was more pronounced in sales of minority player cards."
> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.
It sounds like this study was using real-world applicants, and the other study they're comparing against was using synthetic applicants.
Consider the chance of being accepted as being composed of signal+bias+noise. Noise is random. Signal is a per-applicant value, and what's meant to be measured. Bias is a per-group value, and an artifact of the measuring process.
If acceptance/rejection is independent between positions applied for (as in the synthetic applicant study), that suggests that it's random or composed entirely of noise; ie there is no signal; ie the applicants are all equally qualified.
If acceptance/rejection is correlated, that means there is some nonzero amount of (signal+bias). But real-world applicants are not all identical, so there should be some amount of signal. So you can't just assume zero signal in order to infer that there must be bias.
A inferior candidate (by skill) is going to be consistently rejected, no?
:)
Screened Applications [13]
Unscreened Applications [39148]My point is that this is effectively an auto rejection
Only 40% self report gender/race
no resume data, no education information, degrees, schools, GPA, major, work experience, skills/certifications
Zero job qualifications
> 30% of Black applicants apply to at least one position that demonstrates adverse impact against Black applicants.
The whole thing reads like a tautology.
> We follow 3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors. Each job application was assessed by an AI hiring tool built by a single third-party vendor.
3.4 million people applying to just 150 employers... Who are all using just 1 platform. WTF. This is where the discrimination is happening. Why the f do 3.4 million people feel forced to apply to just 150 employers and why the f do all these 150 employers feel forced to use just one platform. WTF.
Even if that's just part of each company's total hiring pipeline, it's clear; something's wrong. I don't know how long this study has been running but 22k is a lot of people, even over a year. These companies are too big. That's the problem.
>If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring.
I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, was considered at the end.
For the AI study real data from "3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors" was used.
I would be surprised if the results were different.
https://www.yahoo.com/news/us/articles/california-judge-upho...
Sounds a bit like Simpson's Paradox [0]
AI works by learning patterns. So it will become bias by just learning from factors like education history, schools attended, employment history, ZIP codes, or geographic location. Those 3 factors alone are an easy proxy for race.
And if you add names into the equation (if the AI was trained without removing applicant names), the model can become even more bias.
I see nothing that shows any system was making a decision on race. How is the race being presented to the AI?
All this is showing from what I can see, is that certain groups of people were more often denied a next step in the process - but why?
Was the AI going by spelling and grammar? Were there names that were different but the rest of the resume was exactly the same? Were there pictures?
There were mentions that the rate of each group may be more prominent in the data when you split apart different types of jobs instead of all jobs in aggregate.. One could read that like it's inferred; that more warehouse jobs are offered to a race and less admin jobs.. but that same would happen if AI was more focused on perfect grammar for one job and it was not as much of a factor for a warehouse job.
Also if the people applying for the various jobs were self selecting, acceptance percentages this would skew things based upon which ones were applied / not applied to right?
There are so many ways you could draw conclusions like this from data, however correlation is not causation, yet this seems to say it is.
I feel this is an important thing to watch, but Stanford may not be the place to trust with 'Policy Recommendations' as it's very unclear there is any proof that 'AI Hiring Tools Yield Racial Bias and Systemic Rejection' from this study and paper.
PS - now that I see the HN title did not have the word "can" in it, and the title of the article is actually "Tools Can Yield" - maybe that is less accusing and more noting.
I guess this one just compounds.
Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.
The phrase "most-favored" means, "most recommended by the AI relative to the field".
What did you think this sentence meant?
The paper's conclusion, that we need to study this more, is showing the authors likely believe this to be a byproduct of inherent/invisible bias.
Hypothetical SAT score: 1060
How does that help you predict the race of an individual applicant? It's been a while since I took the SAT, but I didn't realize one's score provided so much information.
(I assume they're just using a big LLM for this, it doesnt say, it just says "AI" when they say "AI like that they usually mean LLM".. A custom trained hiring ML system would be better)
Happy to share some sample reports if anyone is interested!
Too many of these studies only focus on percentages and the end result is unqualified candidates getting hired from minority groups at the expense of qualified ones.
The authors are saying it's worth doing more research, because in a controlled data set the results appear unbalanced.
Looks like you didn't read the paper. There are no resumes involved. It is about assessment games.
I tried it before, and discrimination is there, I would get one resume rejected quickly and few days later the same company would invite another resume for a screening call. I tried this before and after AI hype, results weren’t that different btw, and that was tested in US and Canada employers only.
> There is no such thing as anti-white racism.
If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is. Racism is pervasive, insidious and systemic.
A good example in the hiring space is what's called the "second syllable name problem". Traditionally Afrcian names often stress the second syllable (eg Jamal, Lakisha, Malik, Lashonda). Studies have shown that such names have higher rejection rates in job applications [1]. So if you're wondering about the four-fifths rule, it's because it exposes this kind of bias. It's not proof of bias. It simply means further investigation is required.
The problem with AI hiring tools is the logic is opaque. You have no idea why an AI system is rejecting or selecting candidates and you may find it's doing something illegal. Some companies want to hide behind this opaqueness, arguing that if no explicit decision was made then there is no bias. But that's not how system racism works.
There are many such signals that correlate with race that if they affect selection rate, it could be a problem. Did you go to an HBCU? Was your high school in a minority-majority area? What about your previous employers?
This kind of bias doesn't have to be intentional.
[1]: https://www.npr.org/2024/04/11/1243713272/resume-bias-study-...
> If you find yourself wanting to disagree with that then, I'm sorry but you simply don't know what racism is.
You are saying that if you think anti-white racism can exist, you don't know what racism is. That's obviously ludicrous.
The colloquially version, which means "prejudice based on race" and a second version, which specific groups and people have advocated for, which means something like "structural oppression through cultural and governmental means". It's more complicated than just that, but it's a fairly narrow term for them.
So when one person says "there's no such thing as anti-white racism", you hear, "No one's prejudiced against white people for being white!" Obviously, that's ludicrous.
But that person is likely using the, I have no idea what to call it, "advocate definition" maybe, definition would which preclude anti-white racism from existing within that narrow definition of racism.
So it's a debate where people aren't speaking the same version of a language, convinced each other are uninformed, reactionary or stupid.