> The part that's concerning about ChatGPT is that a computer program that is "confidently wrong" is basically indistinguishable from what dumb people think smart people are like.
I don't know, the program does what it is engineered to do pretty well, which is, generate text that is representative of its training data following on from input tokens. It can't reason, it can't be confident, it can't determine fact.
When you interpret it for what it is, it is not confidently wrong, it just generated what it thinks is most likely based on the input tokens. Sometimes, if the input tokens contain some counter-argument the model will generate text that would usually occur if an claim was refuted, but again, this is not based on reason, or fact, or logic.
ChatGPT is not lying to people, it can't lie, at least not in the sense of "to make an untrue statement with intent to deceive". ChatGPT has no intent. It can generate text that is not in accordance with fact and is not derivable by reason from its training data, but why would you expect that from it?
> Based on my understanding of the approach behind ChatGPT, it is probably very close to a local maximum in terms of intelligence so we don't have to worry about the fearmongering spread by the "AI safety" people any time soon if AI research continues to follow this paradigm.
I agree here, I think you can only get so far with a language model, maybe if we get a couple orders of mangitude more parameters it magically becomes AGI, but I somehow don't quite feel it, I think there is more to human intelligence than a LLM, way more.
Of course, that is coming, but that would not be this paradigm, which is basically trying to overextend LLM.
LLMs are great, they are useful, but if you want a model that reasons, you will likely have to train it for that, or possibly more likely, combine ML with something symbolic reasoning.