Hacker News Reader

Comment by jacobjjacob | Hacker News Reader

I think what an LLM is best at is fooling people into thinking it’s intelligent. It is really good at saying things in a natural sounding way, and statistically often getting it right, because certain strings of tokens are encoded. But it’s clear when you start poking that it just as easily tells you that 5/2=3, or that 2+2 != 4. it doesn’t model math or any sort of knowledge at all.

View on HN

Something that I don't quite understand is why the tendency of ChatGPT to be inaccurate sometimes is a fundamental flaw rather than something that can be improved on iteratively if its just a matter of improving the statistical likelihood of accuracy. The question of whether its AGI or not is, to paraphrase the famous quote, a bit like the question of whether a submarine can swim.

Because ChatGPT isn't thinking. It's not reasoning at all. It's assembling sentences that are statistically predicted from using existing writings as the template.

Accuracy isn't a part of the process except in terms of how accurate the training data is. ChatGPT is not making any sort of truth or accuracy determination, let alone doing so poorly.

Can you point to where the thinking happens in a human?

Nope. But I don't have to do that to understand that LLMs do not assess truth or accuracy of anything.

But you don't know that humans don't reason by stringing words together and seeing how statistically likely they seem.

Related to this topic, see "Babble and Prune": https://www.lesswrong.com/s/pC6DYFLPMTCbEwH8W

I don’t think it’s a stretch to say humans aren’t great at assessing the truth or accuracy of anything either.

My point isn't about how good or bad this is being done. Humans, at least some of the time, attempt to assess truth and accuracy of things. LLMs do not attempt to do this.

That's why I think it's incorrect to say they're bad at it. Even attempting it isn't in their behavior set.

Where is the organ that does that? My impression is everything the brain does is homomorphic to what LLMs do.

We don’t have any proof of that, as we don’t have proof also about its opposite. We have no idea why neural nets work, or how our brain works in context of this. There is definitely something human-like in neural networks, but we don’t have any idea why, and what exactly. It’s a completely empirical field and not a theoretical one. We don’t have any good idea what would happen if we could built a 180 billion neuron large neural net, because there is no theory which would prove what would happen even the current ones. That’s why I’ve seen almost every single prediction about what AI would solve in the following years in the past 40 years fail. We have no clue.

There is research that shows humans are also predicting the next word.

I posted that here: https://news.ycombinator.com/item?id=34875324

Isn’t this the whole point of John Searl’s “the Chinese room” thought experiment? But does it matter what is actually going on inside the room, if the effect and function is indistinguishable? Edit: after conferring with ChatGPT, Searle’s point like yours is that the man in the room doesn’t understand Chinese, he is just manipulating symbols, but from the outside, the man in the room seems to speak fluent Chinese.

The whole point of using a computer instead of doing something yourself is to do something quickly and accurately. If I need someone to give me maybe-correct-maybe-not information, I'll just ask one of my coworkers.

Well, that was the point of computers until now. That doesn't mean computers can't be other things, too. ChatGPT is a lot cheaper and faster than your coworker, and it's available (almost) 24 hours a day. And the accuracy may improve!

I mean only if you want accurate information, but if you're building a misinformation network of bots to cause problems in an enemy state then a human sounding bullshit machine sounds like something any number of governments would buy into.

I think a better analogy is asking if a water bottle can swim. It floats most of the time, and can move around if pushed.

The reason “can be inaccurate sometimes” is a fundamental flaw is because my assumption is that it will never not be inaccurate. I think it will always be inaccurate sometimes and never be accurate always.

This doesn’t mean it isn’t useful for a lot of applications. But I don’t think it is a holy grail technology, it’s not AGI, and it isn’t going to replace professions.

The best feature of this LLM is that it goes from fooling people into having people make fools of themselves when they turn around and predict the end of the world/education/programming/whatever thing they don't quite understand based on what a confidently incorrect charlatan machine told them. It's like a viral marketing gag.

I could say the same things about some of my coworkers...

Fair point, I think we would also call them frauds.

Reply on news.ycombinator.com