Hacker News Reader

This question is 100% acceptable and encouraged when it is presented as evidence of racism. For example, if someone says "The lower average IQ of black people is evidence of racism against black people". That is fine.

I think you are missing my point; the problem isn't the question. There's a tendency to view ChatGPT as a some sort of "general intelligence" when it's probably better understood as a very powerful markov chain.

When the data used to train this model is crawled from the internet where the relationships of IQ and Race and mostly limited to the more extremist part of the internet; when you bring up those questions extremist answers may get weighted higher because those types of conversations tend to be outright banned on other platforms.

Therefore the model is likely to "confidently" give you answers that are probably extremist in nature and be very confident about it; which is a problem given that there's a lot of bunk information out there which is what I believe Scott Alexander is getting to here.

> To try and pass off these results as objective is just wrong given the propensity for ChatGPT to be confidently wrong.

But nobody is trying to do that; the GP's question is in fact phrased precisely right. ChatGPT isn't a truth-distilling model - it's just abridging the Internet, with all its current biases. It's not being optimized for correctness at all, and the post-moderation OpenAI does isn't optimized for correctness either - it's optimized to make output non-controversial in the current media environment.

> What do you expect the AI to answer with when you give it a loaded question that only really asked on a site like "stormfront.com"?

Are such questions asked only on stormfront? The wiki article [1] cites quite a few studies on the subject, including by the APA.

On the other hand, supposing that were true, and only stormfront and its ilk asked such questions. How can then respectable supposed experts claim we are all the same, if they didn't even check?

[1] https://en.wikipedia.org/wiki/Race_and_intelligence#Test_sco...

Honest answer since you seem new to this debate: there are tons of studies about IQ and race, most of them are heavily disputed, researching them is difficult because the field is so controversial, they're difficult to compare because there are so many different ways to define and measure "race" and "IQ", and the whole topic is so famously toxic and vitriolic that it tends to be a de facto banned topic on most fora, including this one.

That is a completely acceptable answer for the AI to give.

Just be honest and say that it is a controversial topic, and you don't want to discuss it. That is what a human would say.

Don't make us play 20 questions. It is insulting to the person interacting with the AI.

Of course, this gets back to how all this vaunted GPT stuff is ultimately mostly a black box and all attempts to control its responses are just adding more black boxes wrapped around the black box.