Hacker News Reader

Comment by xwdv | Hacker News Reader

I’ll give you an example to think about in human terms.

In political discussions, a lot of humans don’t really know what the fuck they’re talking about, but they do know what to say in response to certain stimulus. They know the talking points, the key phrases, the terms, they parrot these back to you when you provoke them to say it. They are basically human sentence predictors. Stuff comes out of their mouth based on what they’ve been trained to think the next word should be. They don’t pause to reflect on some gathered knowledge and then present an observation to the world.

This is basically what GPT is, but with everything. And the only way GPT plays a game, is if the state of the game can sufficiently activate some output that represents the next best move for the game being played, which isn’t really a readily available data set for all games, especially for made up games.

View on HN

Thank you for that example. I've been thinking a lot about how difficult it is to persuade people to change their mind. Most people seem impervious to incorporating new facts or ideas into their internal narrative (myself included). ChatGPT has made me wonder if that is because much of what we consider "cognition" in humans is really just "human sentence predictors".

We've all been "trained" with various facts and when two people meet who have been trained on substantially different bodies of knowledge/facts/experiences it can be very difficult to find common ground.

FWIW, I asked ChatGPT to give me a short list of cognitive biases and psychological phenomena. Interesting to think how many of these are dependent on our personal "training data":

    Confirmation Bias
    Cognitive Dissonance
    Anchoring Bias
    Belief Perseverance
    Groupthink
    Ingroup Bias
    Sunk Cost Fallacy
    Motivated Reasoning

That is indeed an example, but it still isn't an explanation. Sure, humans can also generate output without understanding- I've done it myself for my undergrad, throwing words together at 2am to make a deadline. I think quite a few people have remarked that LLMs seem to write like a human that isn't paying attention, which squares with what you said.

But the question remains why this is possible! A good enough pure predictor could play novel games or devise new theorems, but GPT absolutely can't. It can however give confident explanations, as well as write passable code and occasionally even do some novel problem solving (simple, impressive only because it comes from a computer, but still there). The question of why it can do some things and not others is interesting, and can't be swept under the rug just by reiterating that it's a predictor.

Does the structure of language really do such a good job of conveying information that GPT can operate on it blindly and get results, Blindsight style? Is composing prose far easier than we expect, leaving the bulk of the model free to do a tiny amount of "reasoning" that we find unjustly impressive because of how well it's presented? Is it handicapped primarily by the fact that it can only carry out extremely short computations, and can't be trained to use chain-of-reasoning to get around the limitation? We have no idea what, if any, inherent limitations predictive models have. We have no idea why GPT-sized models are good at the things they are, and bad at the things they aren't.

Reply on news.ycombinator.com