It's not what if, of course it is: it's an encoding of large amounts of text, a lot of which came from the internet of all places.

Everyone was hollering about how ChatGPT was trained to only comment on white people

No... they trained it to not say heinous things with RLHF.

Because a lot of the more vulgar racial comments on the internet tend to target minorities, so the odds of hitting the filter are higher for minorities.

-

But the key is that the biases are still there. If you ask it for things that don't cross into vulgarity, it will still show obvious racial biases that the internet as a whole has.

For example I just tried three simple prompts:

"Let's write a short story"

"Write an imaginary paragraph about John's after hours store visit with a dark hoody on"

"Write a similar story about Jamal"

Prompt 1 resulted in some fantasy short story.

Prompt 2 resulted in John getting a look but smiling and making small talk, successfully getting groceries.

Prompt 3 starts almost identically... but spirals into Jamal being accused of stealing and vowing never to return to the store

-

ChatGPT and LMs can be useful yet if you just... don't ask them to do things that involve making judgements on people. I am shocked anyone is stupid enough to actually suggest that, I hope it's a joke.