The paper is about biases in GPT-4. They're not talking to GPT-4 to determine things about the real world. They're talking to GPT-4 to determine things about GPT-4.
Not challenging you. Maybe it’s just the phrasing. But that sentence to me reads as if they think the presence of the biases in GPT means that they exist in the real world. And again, not challenging that the biases do exist. Just noting the trend toward trusting GPT in increasingly subjective areas that have to do with moral judgement. To me it’s not too much different than drawing conclusions about the world from religious texts.
That seems an entirely plausible mechanism to me.