> If by default an LLM refers to engineers as "he" more than "she", it's because engineers are more likely to be male than female.
That's a great point to raise, thank you for mentioning it. This really is the crux - machine models crystallize _current_ bias and amplify it. Most programmers in the early days of computers were women, but were displaced over time. Should the training data reflect the bias of the dominance of women in the early days, the dominance of men now, or some other combination?
> Most people would prefer an AI whose model of reality matches actual reality
Training data is not reality; training data is training data. It will fundamentally lag behind reality, and it will have a bias towards past conditions and precedents.
You do realise, if you command an LLM to talk like a caveman it doesn't invent a time machine, travel back in time and research how cavemen actually talked; it just produces the "me hit with rock" that redditors imagine a caveman would have talked like.