LLMs can't perform "genuine logical reasoning," Apple researchers suggest
arstechnica.com
arstechnica.com
And the day before that: https://news.ycombinator.com/item?id=41808683
Understanding the Limitations of Mathematical Reasoning in LLMs - https://news.ycombinator.com/item?id=41808683 - Oct 2024 (266 comments)
Apple study proves LLM-based AI models are flawed because they cannot reason - https://news.ycombinator.com/item?id=41823822 - Oct 2024 (19 comments)
It's an impressive technology but its limits are highly overlooked in the current hype cycle.
AI researchers have known this from the start and won't be surprised by this because it was never intended to be able to do this.
The problem is the customers who are impressed by the human-sounding bot (sounding human is exactly what an LLM is for) and mentally ascribe human skills and thought processes to it. And start using it for things it's not, like an oracle of knowledge, a reasoning engine or a mathematics expert.
If you want to have knowledge, go to a search engine (a good one like kagi) which can be ai assisted like perplexity. If you want maths, go to Wolfram Alpha. For real reasoning we need a few more steps on the road to general AI.
This is the problem with hypes. People think a tech is the be all end all for everything and no longer regard its limitations. The metaverse hype saw the same problem even though there's some niche usecases where it really shines.
But now it's labelled as a flop because the overblown expectation of all the overhyped investors couldn't be met.
What an LLM is great at is the human interaction part. But it needs to be backed by other types of AI that can actually handle the request and for many usecases this tech still needs to be invented. What we have here is a toy dashboard that looks like one of a real car, except it's not connected to one. The rest will come but it'll take a lot more time. Meanwhile making LLMs smarter will not really solve the problem that they're inherently not the tool for the job they're being used for.
Sure, but it doesn’t help when well-respected researchers, like Ilya Sutskever, go around talking about how OpenAI’s LLMs have intelligence. There have been plenty of commenters on HN who, without a hint of irony, talk about how “well maybe self-attention is the mechanism of consciousness”. And all the scaling papers suggest (suggest in the sense that they seem to want to draw that inference, not that I am endorsing it) that LLMs have no limit to scaling with more parameters and training tokens. Still other (serious, and well-cited) papers run benchmarks that include math and logic tests… why would anyone do that if they genuinely believed that LLMs are just stochastic next-token predictors?
So this is a little more than “you’re holding it wrong”. The entire AI/ML industry is telling you to hold it that way, then acting shocked when they discovered gambling in the establishment.
This is the key issue, I think.
> it was never intended to be able to do this
But to be fair, plenty of technologies start off that way until someone finds a way to make the technology do something it was never intended to do
It made some billionaires who will argue it was a tremendous idea. But in the long term I think it will cause another AI winter that will dry up funding for useful research that would take longer to mature. Or maybe it's just like fusion... promising on paper but so incredibly expensive to handle as to render it useless in practice.
LLMs are extremely useful but it's important to recognize the limited use cases:
- summarizing information
- faster search (summarizes search results for you)
- writing in fluent English
- translating between common languages
- realistic sounding chatbots, customer support
- some lower level coding tasks
- technical answers from published documentation - RTFM for you
None of these are any more "AI" than the previous cycle of "AI" technology
Woah neat I've been wondering about this myself, any reading you'd recommend?
The Stack: On Software and Sovereignty https://direct.mit.edu/books/monograph/3504/The-StackOn-Soft...
The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power https://www.hbs.edu/faculty/Pages/item.aspx?num=56791
Platform Capitalism https://www.wiley.com/en-in/Platform+Capitalism-p-9781509504...
After all, nature can do it with 4kg of brain mass and less than 100 Watts. There must be a way we can do this in tech too.
But yeah I doubt it would be a benefit for society too.
The same way that openai did what they did using the whole of common crawl, god knows how many phds, decades of development of basic concepts and several years of datacenters worth of compute. And they got a chatbot.
ChatGPT has the branding and first mover moat. Outside of tech folks, Claude/Gemini/Mistral/Phind/Perplexity/Midjourney do no exist, only ChatGPT+CoPilot are real and AI.
Rot13 meaning that LLMs can't do Rot 3, 4, ..., n except for Rot13 (because that' in the training data)
Mystery Blocks World being a trivial "translation" (by direct replacement of terms) of a simple Blocks World. The LLMs can solve the original, but not the "translation" - susprisingly, even when provided with the term replacements!
Both are discussed in Prof. Subbarao's Machine Learning Street Talks episode
What’s funny is that AI is now being trained by a human accepting or rejecting its answers, probably not on the basis of the rigor of the answer since the temp worker hired to do it is probably not a logician, mathematician, or scientist. I suspect most people’s reasoning is closer to an LLM’s than we would be comfortable admitting.
1. The vast vast majority of the time
If abduction has an Achilles Heel, it's that the sequence of reasons proposed is only ONE OF MANY possible chains of reasons that could have led to the current outcome. That tells you nothing about one possible chain's correctness and little about its likelihood to be correct (since you need to understand the degrees of freedom in each scenario to do that). That makes abduction more useful as a means to express personal bias and manipulate the beliefs of others than an attempt to fair-mindedly understand how something came to be.
It's hardly surprising that LLMs employ abduction rather than deduction or induction. Abduction is really just storytelling, where you recount a similar tale that resembles the current scenario (and ends where you want it to). It doesn't require any ability to generalize similar events into precise logical rules based on instances that share the same dependencies, mechanism of action, and outcome. Only by creating such rules using induction can deductive reasoning then take place using them. But LLMs associate, they don't equate. To date, I see no path to that changing.
In other words, ChatGPT continues to dominate. A 0.3% drop might as well be noise.
Also the original, allegedly more expensive GPT-4 (can we call it ChatGPT-4og??) is conspicuously missing from the report...
Instead of the dead Internet theory, we should start finding what percent of the population is no better than a LLM.
The real fun is in intellectual engagement, but if thread is generated by bots and commented by bots as well, all I can see is a fake depiction of activity.
However, I understand that, my perspective of beneficial activity could be limited.
And dont forget, reddit started life as a bunch of owner sock puppets.
Also they mapped an insect brain
Seems like my several comments suggesting AI scientists should peek other fields, did get some attention.
That probably makes me the most talented and insightful AI scientist on the planet.
Comparing one-shot LLM responses with what a human can do in their head doesn’t make much sense. If you ask a person, they would try to work out the answer using a logical process but fail due to a shortage of working memory.
An LLM will fail at the task because it is trying to generate a response token by token, which doesn’t make any sense. The next digit in the number can only be determined by following a sequence of logical steps, not by sampling from a probability distribution of next tokens. If the model was really reasoning the probability for each incorrect digit would be zero.
https://x.com/yuntiandeng/status/1836114401213989366
If chain of thought really worked we should see no difference between 1 digit and 20 digit multiplication.
ChatGPT 4o as of right now just runs python code, which I guess is "Let me get my calculator", see https://chatgpt.com/share/670df313-9f88-8004-a137-22c302f8bf...).
Claude 3.5 just... does the multiplication correctly by independently deciding to go step-by-step (don't see a convenient way to share conversations, but the prompt was just "What is 1682671* 168363?").
in other words, for the LLMs that do that kind of thing well, like gpt-o1, don't they essentially also use 'a pen and paper'?
Ask LLMs without chain of thought built-in is the same as to ask people to multiply these numbers without pen and paper. And LLMs with chain of thought actually are capable of doing this math.
If you tell an LLM to explain how to multiply two numbers it will give a flawless textbook answer. However when you ask it to actually multiply the numbers it will fail. LLMs have all the knowledge in the world in their memory, but they can't connect that knowledge into a coherent picture.
Do you think your inner monologue is any different? Because it sure as hell isn’t the same system as the one doing math, or recognising faces, or storing or retrieving memories, to name a few
And chain of thought is kind of like giving that brain some scratch space to figure out the problem.
This simulated brain can't access multiplication instructions on the CPU directly. It has to do the computation via it's simulated neurons interacting.
This is why it's not so surprising that this is an issue.
The level of detail of the simulation has little bearing on this. And in fact whether you call it a simulation or something else doesn't matter either. Understanding that the LLM does not compute by using the CPU or GPU directly is what's necessary to understand why computation is hard for LLMs.
I don’t know, that’s why I ask.
I mean, what is being described seems like super basic debug step for any real world system. This is kind of stuff not very advanced QA teams in boring banks do to test your super-boring not very advanced back-office bookkeeping systems. After this kind of testing reveals a number of bugs, you don't erase this bookkeeping system and conclude banking should be done manually on paper only, since computers are obviously incapable of making correct decisions, you fix these problems one by one, which sometimes means not just fixing a software bug, but revisioning the whole business-logic of the process. But this is, you know, routine.
So, not being aware of what are these benchmarks everyone uses to test LLM-products (please note, they are not testing LLMs as some kind of concept here, they are testing products), I would assume that OpenAI in particular, and any major company that released their own LLM product in the last couple of years in general, already does this super-obvious thing. But why this huge discovery happens now, then?
Well, obviously, there are 2 possibilities. Either none of them really do this, which sounds unbelievable - what all these high-paid genius researchers even do then? Or, more plausibly, they do, but not publish that. This one sounds reasonable, given there's no OpenAI, but AltmanAI, and all that stuff. Like, they compete to make a better general reasoning system, of course they don't want to reveal all their research.
But this doesn't really look reasonable to me (at least, at this very moment) given how basic the problem being discussed is. I mean, every school kid knows you shouldn't test on data you use for learning, so to be "peeking into answers when writing a test" only to make your product to perform slightly better on popular benchmarks seems super-cheap. I can understand when Qualcomm tweaks processors specifically to beat AnTuTu, but trying to beat problem-solving by improving your crawler to grab all tests on the internet is pointless. It seems, they should actively try to not contaminate their learning step by training on popular benchmarks. So what's going on? Are people working on these systems really that uncreative?
This said, all of it only applies to general approach, which is to say it's about what article claims, not what it shows. I personally am not convinced.
Let's take kiwi example. The whole argument is framed as if it's obvious that the model shouldn't have substracted these 5 kiwies. I don't know about that. Let's imagine, this is a real test, done by real kids. I guarantee you, the most (all?) of them would be rather confused by the wording. Like, what should we do with this information? Why was it included? Then, they will decide if they should or shouldn't substract the 5. I won't try to guess how many of them will, but the important thing is, they'll have to make this decision, and (hopefully) nobody will suddenly multiply the answer by 5 or do some meaningless shit like that.
And neither did LLMs in question, apparently.
In the end, these students will get the wrong answer, sure. But who decides if it's wrong? Well, of course, the teacher does. Why it's wrong? Well, "because it wasn't said you should discard small kiwies!" Great, man, you also didn't tell us we shouldn't discard them. This isn't a formal algebra problem, we are trying to use some common sense here.
In the end, it doesn't really matter, what teacher thinks the correct answer is, because it was just a stupid test. You may never really agree with him on this one, and it won't affect your life. Probably, you'll end up making more than him anyway, so here's your consolation.
So framing situations like this as a proof that LLM gets things objectively wrong just isn't right. It did subjectively wrong, judged by opinion of Apple researchers in question, and some other folks. Of course, this is what LLM development essentially is: doing whatever magic you deem necessary, to get it give more subjectively correct answers. And this returns it's to my first point: what is OpenAI's (Anthropic's, Meta's, etc) subjectively correct answer here? What is the end goal anyway? Why this "research" comes from "Apple researchers", not from one of these compenies' tech blogs?