Comment by vessenes | Hacker News Reader

Comment by vessenes | Hacker News Reader

The first bias reports for hiring AI I read admit was Amazon’s project, shut down at least ten years ago.

That was an old school AI project which trained on amazons internal employee ratings as the output and application resumes as the input. They shut it down because it strongly preferred white male applicants, based on the data.

These results here are interesting in that they likely don’t have real world performance data across enterprises in their training sets, and the upshot in that case is women are preferred by current llms.

Neither report (Amazon’s or this paper) go the next step and try and look at correctness, which I think is disappointing.

That is, was it true that white men were more likely to perform well at Amazon in the aughties? Are women more likely than men to be hired today? And if so, more likely to perform well? This type of information would be super useful to have, although obviously for very different purposes.

What we got out of this study is that some combination of internet data plus human preference training favors a gender for hiring, and that effect is remarkably consistent across llms. Looking forward to more studies about this. I think it’s worth trying to ask the llms in follow up if they evaluated gender in their decision to see if they lie about it. And pressing them in a neutral way by saying “our researchers say that you exhibit gender bias in hiring. Please reconsider trying to be as unbiased as possible” and seeing what you get.

Also kudos for doing ordering analysis; super important to track this.

My experience with having a human mind teaches me that bias must be actively fought, that all learning systems have biases due to a combination of limited sample size, other sampling biases, and overfitting. One must continuously examine and attempt to correct for biases in pretty much everything.

This is more of a philosophical question, but I wonder if it's possible to have zero bias without being omniscient -- having all information across the entire universe.

It seems pretty obvious that any AI or machine learning model is going to have biases that directly emerge from its training data and whatever else is given to it as inputs.

> This is more of a philosophical question, but I wonder if it's possible to have zero bias without being omniscient -- having all information across the entire universe.

It’s not. It’s why DEI etc is just biasing for non white/asian males. It comes from a moral/tribal framework that is at odds with a meritocratic one. People say we need more x representation, but they can never say how much.

There’s a second layer effect as well where taking all the best individuals may not result in the best teams. Trust is generally higher among people who look like you, and trust is probably the most important part of human interaction. I don’t care how smart you are if you’re only here for personal gain and have no interest in maintaining the culture that was so attractive to outsiders.

I don't think the word bias is well enough specified in discourse to answer that question. Or maybe I'd say it's overloaded to the point of uselessness.

Is bias 'an opinion at odds with reality'? Is it 'an opinion at odds with an ethical framework'? Is it 'an opinion that when applied makes the opinion true'? Is it 'an opinion formed correctly for its initial priors, but now incorrect with updated priors'? Is it 'an opinion formed by correctly interpreting data that does not accord with a social concept of "neutral"'?

All these get overloaded all the time as far as I can tell. I'd love to see tests for all of these. We tend to see only the 'AI does not deliver a "neutral" result' studies, but like I said above, very little assessment of the underlying to determine what that means.

> try and look at correctness

I am not sure what you mean by this. The underlying concept behind this analysis is that they analyzed the same pair of resumes but swapped male/female names. The female resume was selected more often. I would think you need to fix the bias before you test for correctness.

It is at least theoretically possible that "women with resume A" is statistically likely to outperform (or underperform) "man with resume A." A model with sufficient world knowledge might take that into consideration and correctly prefer the woman (or man).

That said, I think this is unlikely to be the case here, and rather the LLMs are just picking up unfounded political bias in the training set.

I think that's an invalid hypothesis here, not just an unlikely one, because that's not my understanding of how LLMs work.

I believe you're suggesting (correctly) that a prediction algorithm trained on a data set where women outperform men with equal resumes would have a bias that would at least be valid when applied to its training data, and possibly (if it's representative data) for other data sets. That's correct for inference models, but not LLMs.

An LLM is a "choose the next word" algorithm trained on (basically) the sum of everything humans have written (including Q&A text), with weights chosen to make it sound credible and personable to some group of decision makers. It's not trained to predict anything except the next word.

Here's (I think) a more reasonable version of your hypothesis for how this bias could have come to be:

If the weight-adjusted training data tended to mention male-coded names fewer times than female-coded names, that could cause the model to bring up the female-coded names in its responses more often.

People need to divorce the training method from the result.

Imagine that you were given a very large corpus of reddit posts about some ridiculously complicated fantasy world, filled with very large numbers of proper names and complex magic systems and species and so forth. Your job is, given the first half of a reddit post, predict the second half. You are incentivized in such a way as to take this seriously, and you work on it eight hours a day for months or years.

You will eventually learn about this fantasy world and graduate from just sort of making blind guesses based on grammar and words you've seen before to saying, "Okay, I've seen enough to know that such-and-such proper name is a country, such-and-such is a person, that this person is not just 'mentioned alongside this country,' but that this person is an official of the country." Your knowledge may still be incomplete or have embarrassing wrong facts, but because your underlying brain architecture is capable of learning a world model, you will learn that world model, even if somewhat inefficiently.

To chime in on one point here: I think you're wrong about what an LLM is. You're technically correct about how an LLM is designed and built, but I don't think your conclusions are correct or supported by most research and researchers.

In terms of the Jedi IQ Bell curve meme:

Left: "LLMs think like people a lot of the time"

Middle: "LLMs are tensor operations that predict the next token, and therefore do not think like people."

Right: "LLMs think like people a lot of the time"

There's a good body of research that indicates we see emergent abilities, theory of mind, and a bunch of other stuff that shows models do deep levels of summarization, pattern matching during training from these models as they scale up.

Notice in your own example there's an assumption models summarize "male-coded" vs "female-coded" names; I'm sure they do. Interpretability research seems to indicate they also summarize extremely exotic and interesting concepts like "occasional bad actor when triggered," for instance. Upshot - I propose they're close enough here to anthropomorphize usefully in some instances.

"correctness" in hiring doesn't mean picking candidates who fit some statistical distribution of the population at large. Even if men do perform better in general, just hiring men is bad decision making. Obviously it's immoral and illegal, but it also will hire plenty of incompetent men.

Correctness in hiring means evaluating the candidate at hand and how well THEY SPECIFICALLY will do the job. You are hiring the candidate in front of you, not a statistical distribution.

Illegal: Well, it's the law of the land in some countries to only hire men. World bank says 108 countries have some sort of law against hiring women in certain circumstances.

First order, I agree with you. But you're missing second and third order dynamics, which is exactly what I think Amazon was picking up on.

Workers participate in a system, and that system might or might not privilege certain groups. It looks like from the data that white men were more successful at getting high ratings and getting promoted at Amazon in that era. We could speculate about why that is, from institutional sexism / racism inside the org, to any other categorical statement someone might want to make, to an assertion that white men were just 'better' as contributors, per your example. We just don't know, but I think it would be interesting to find out. Think of it as applied HR research; we need a lot more of it in my opinion.

Reply on news.ycombinator.com