Real Signals or Artificial Stereotypes?

> I’d created 2000 free-text responses and labelled them ‘UK’. Then I copied and pasted the exact same 2000 responses but labelled these ‘US’. Finally, I combined them to create a dataset of 4000 total responses, and jumbled them up.

> Despite the responses being identical for the UK and US, Copilot produced a rich, detailed summary of how US and UK respondents differed.

I'll bet there are lots of people trying to use llms to "discover" differences in datasets and just seeing whatever biases were in the base training data instead. Good that someone is out there showing how badly it can go, though I doubt the people who need to see it will :-/