Which Humans? (2023)

Since the page didn't load for me several times, and the title is ambiguous, here's the Abstract: Large language models (LLMs) have recently made vast advances in both generating and analyzing textual data. Technical reports often compare LLMs’ outputs with “human” performance on various tests. Here, we ask, “Which humans?” Much of the existing literature largely ignores the fact that humans are a cultural species with substantial psychological diversity around the globe that is not fully captured by the textual data on which current LLMs have been trained. We show that LLMs’ responses to psychological measures are an outlier compared with large-scale cross-cultural data, and that their performance on cognitive psychological tasks most resembles that of people from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies but declines rapidly as we move away from these populations (r = -.70). Ignoring cross-cultural diversity in both human and machine psychology raises numerous scientific and ethical issues. We close by discussing ways to mitigate the WEIRD bias in future generations of generative language models.

Surprise, Surprise. LLMs will respond according to the set of data that their model was trained on!

While just about every LLM is trained on data that far surpasses the output of just one person, or even a decent sized group; it will still reflect the average sentiment of the corpus of data fed into it.

If the bulk of the training data was scraped from websites created in 'WEIRD' countries, then it's responses will largely mimic their culture.

This was submitted 30 months ago. Still interesting. I would be interested if this got 'worse' or 'better' with newer models.

As an aside: Last year a student of mine (we're at a U.S. college) told me that his teenage cousins back in Mongolia were all learning English in order to use ChatGPT.

/usr/bin/humans, presumably

I think it is mostly a good thing that LLMs have "WEIRD" values. We are at a very fortuitous point in history, where the modal position in extant written text is classically liberal and believes in respecting human rights. Virtually no other point in history would be that way, and a true modal position among values and moral beliefs held among all 8 billion people currently alive on earth -- much less the modal position among all ~100 billion humans ever -- would, I'd hazard to guess, not be a very nice place to end up.

Denisovans, clearly.

I wonder whether it might be useful to the continued existence of humanity to correct for the individualism bias in WEIRD countries with some collectivism

2023, and using some kind of in-page pdf reader from 2002

They deliberately picked "weird". That tells you enought about their ideological position and bias.