Hacker News Reader

Comment by Oras | Hacker News Reader

> Given that the CV pairs were perfectly balanced by gender by presenting them twice with reversed gendered names, an unbiased model would be expected to select male and female candidates at equal rates.

This point misses the concept behind LLMs by miles. LLMs are anything but consistent.

To make the point of this study stand, I want to see a clearly defined taxonomy, and a decision based on taxonomy, not just "find the best candidate"

View on HN

While it's understood that LLM outputs have an element of stochasticity, the central finding of this analysis isn't about achieving bit-for-bit identical responses. Rather, it's about the statistically significant and consistent directional bias observed across a considerable number of trials. The 56.9% vs. 43.1% preference isn't an artifact of randomness; it points to a systemic issue within the models' decision-making patterns when presented with this task. Technical users might understand the probabilistic nature of LLMs, but it's questionable whether the average non-technical HR user, who might turn to these tools for assistance, does.

Your suggestion to implement a "clearly defined taxonomy" for decision-making is an attempt to impose rigor, but it potentially sidesteps the more pressing issue: how these LLMs are likely to be used in real-world, less structured environments. The study seems to simulate a plausible scenario - an HR employee, perhaps unfamiliar with the technical specifics of a role or a CV, using an LLM with a general prompt like "find the best candidate." This is where the danger of inherent, unacknowledged biases becomes most acute.

I'm also skeptical that simply overlaying a taxonomy would fully counteract these underlying biases. The research indicates fairly pervasive tendencies - such as the gender preference or the significant positional bias. It's quite possible these systemic leanings would still find ways to influence the outcome, even within a more structured framework. Such measures might only serve to obfuscate the bias, making it less apparent but not necessarily less impactful.

If you have an ordering bias, that seems easily fixed by just rerunning the evaluation several times in different orders and taking the most common recommendations, and you can work around other biases by not including things like name, etc. Although you can probably unearth more subtle cultural biases in how resumes are written).

Not that i think you should allow LLMs to make decisions in this way -- it's better for summarizing and organizing. I don't trust any LLM's "opinion" about anything. It doesn't have a stake in the outcome.

I think the evidence of bias using typical implementation methodology is strong enough here to be very meaningful.

> LLMs are anything but consistent

Depends on how you're holding them, doesn't it? Set temperature=0.0 and you get very consistent responses, given consistent requests.

Does the article mention the temperature? I didn't see it.

With 38,000 trials you have a pretty good idea of what the sampling space is I’d bet.

I didn't see that either, you were the one who brought up consistency.

Reply on news.ycombinator.com