This is one of the least interesting questions to ask LLMs. I wish it wasn't so popular.
I don't think generative AI will ever be able to reach AGI, and most people selling LLM today pretend it is AGI
>most people selling LLM today pretend it is AGI
Who are these "most people"?
The choice just comes with trade-offs in memory usage, compute, and effectiveness of the model in various scenarios.
So what we ended up with was a pragmatic/engineering decision rather than a theoretical or fundamental constraint.
LLMs can't effectively execute algorithms similarly in their context, nor can they memorize new data or facts it was given without providing it tools like function calling or embeddings. If you give LLMs tool calling and storage mechanisms then counting letters in words becomes pretty damn reliable.
No one said that LLMs aren’t good for anything.
I pointed out — in response to another poster downplaying mention of a well-known and undisputed limitation that LLMs often have — that it is valid to consider these well-known and undisputed limitations if one is considering using them for anything other than a toy.
It is downright silly to discourage discussion of well-known and undisputed limitations! The only reason for that can only be entirely emotional as there is genuinely nothing tangible to be gained by being steadfast in silence about a fact that isn’t up for debate.
I took your response to be arguing against a message I'd read to be saying something like the above. Especially when you basically seemed to be saying that limitations like this are important in everything but toy applications. It's uninteresting because it doesn't point towards larger problems with their use in the kind of application they're being used for and are intended for unlike prompts that point to weakness in logic or propensity to hallucinate.
Don’t need to be exact as firing is statistical, just give us a good average.
Hard? You can’t count?
Computers count pixels no problem. So weird you can’t.
Dementia? Not an AGI? /h
—-
This is what is happening.
Here are the “Reasons”.
In your vision system, the raw information from individual retina signals is munged into a different representation before reaching a level where you have flexible processing.
Likewise, in LLMs, letters are munged into tokens before LLMs “see” them.
When they sometimes get that “simple” question right, it’s actually a bit of an amazing feat. Given how they are constructed.
—-
Now try counting R’s as you read at a normal rate, or listen to someone speak.
You can’t do that either, during normal processing.
When we add spelling to LLMs training examples, they will do it easily. Just as you learned to do it, only after special lessons, after you had already learned to listen and speak.
Spelling is its own special practiced skill, in humans and LLMs.
This analogy makes sense because everybody could count their red retina cells until a couple years ago when the new painting paradigm arose, and also counting red retinal cells is a good analogy for being able to see simple objects that have always been distinguishable.
It is fascinating how tapping the “Do Not Use LLMs For Computation If The Results Need To Be Reliably Better Than A Random Output” sign invites explanations of why that fact is actually Cool and Good
You learned to listen and speak words before you could spell. Imagine if nobody had actually shown you written words?
Or they were speaking another dialect but expecting you to count R’s in standard English?
LLMs are not trained on words in the form of letters.
They process and generate the words in the form of tokens. Pre- and post-processing systems converts letters to tokens and the reverse, without their ability to access that processing.
Spelling, for both us and LLMs, requires specific training/lessons.
> It is fascinating how tapping the “Do Not Use LLMs For Computation If The Results Need To Be Reliably Better Than A Random Output” sign invites explanations of why that fact is actually Cool and Good
Also fascinating:
People who hallucinate/confabulate ridiculous straw man rationales for people they disagree with, unaware they are filling in gaps in their knowledge regarding other people’s actual reasoning and the actual subject at hand.
So LLM! Such unreliable processing!
Perhaps, start posting a reliability disclaimer?
(There are tasks that LLMs totally fail on that would be obvious to an illiterate human though)
To envision what a next generation model bound by the same constraints should do, it'd be to recognize that it can't count tokens and use code access to write code that solves the strawberry problem without prompting.
Asked to count cells it'd be a model that could write and execute OpenCV tasks. Or to go a step further, be a multimodal model that can synthesize 10000 varations of the target cell, and finetune a model like YOLO on it autonomously.
I find arguments that reduce LLMs to "It can't do the simple thing!!!!" come from people unable to apply lateral thinking to how a task can be solved.
The VQA problems I'm describing can be solved seemingly in one case but not combined with counting. Counting is fundamentally challenging for sort of unknown reasons, or perhaps known to the very best labs who are trying to tackle it directly.
Another POV is that the stuff you are describing is in some sense so obvious that it has been tried, no?
What I described is tried, and works, but the models are still not cheap/fast/reliable enough to always do what I described for every query.
The difference between what I described and directly asking the model to count is that we know the models can get cheaper, faster, and more reliable at what I described without any earth shattering discoveries
Like I don't see any reason why GPT 10 will ever be able to count how many letters there are in the word strawberry without a complete paradigm shift in model building... but going from GPT 3 to GPT 4 we already got a model that can always write the dead simple code required to count it out, and the models that can do so are already getting cheaper and faster every few months without any crazy discoveries.
Maybe they should make you Chief Scientist at OpenAI, I hear the position is open.
Thats about the same kind of question, given that LLMs see tokens instead of characters, and we see characters instead of bits.
GPT 3 would essentially always fail on higher math problems/arithmetic, but whenever I've used it for math GPT 4 has always delegated out to executing code where necessary.
Who exactly does this in this space? Would be good to be able to call them out on it right now.
But it's likely to be an important component in an AGI system. I suppose the interesting question is how to integrate LLMs with more traditional logic and planning systems.
Having said that, the 40b model wasn't able to answer any of my real-world example questions correctly. Some of these (e.g. "how do I add a sequential number after my titles in an HTML page using just CSS, without changing the page") are questions that even some of the better small local models can answer correctly. It gave very authoritatively sounding wrong answers.
That is, all LLMs look at language as a series of opaque, independent tokens, e.g. strawberry might be a single token (say <5678>), or probably two (e.g. straw and berry, say <123><789>). But in no way will it represent it like we will, with letters. So if you ask it "how many R's are in strawberry", it fundamentally can't do any "counting", it just basically has to rely on whether that question (or similar questions about related words) has been asked before so it can predict the next token in its output correctly. I suppose with enough data and LLM could learn to associate character counts with tokens (e.g. with the right training set it could learn metadata about the tokens).
My point being that with only the most basic understanding of how LLMs function, this "character counting" limitation seems blatantly obvious.
LLMs operate with language at multiple levels of abstraction and tokens are not the only way to have character level knowledge.
For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge.
And of course LLMs famously have emergent properties, for which there’s not yet a precise mechanism to illuminate the results.
We’re dealing with very complex systems that are still relatively poorly understood, and I believe the pool of concepts understood to the point of being blatantly obvious is still small.
We know exactly how LLMs tokenize their input - OpenAI even has a public tool you can test out, https://platform.openai.com/tokenizer .
When you say "For example, nothing excludes pre-training data from directly or indirectly encoding such knowledge." - yes, that's why I explicitly said "e.g. with the right training set it could learn metadata about the tokens".
But the point was still put perfectly by another commenter: "How many 1 bits are there in strawberry?" When nearly all humans can't answer that, we're very unsurprised; we see it immediately as a simple difference in how humans encode that string vs. how computers do it. We don't say "Well, the human mind is so complex and filled with emergent properties that the reason for this human limitation is a big mystery". And we also know that if we teach a human the encoding rules for characters (i.e. the bit pattern of each letter), they could answer this question, and similarly that's the analogous thing to training an LLM to learn metadata about the tokens.
Now, what I think is very interesting is why it's so hard to teach an LLM to say "I don't know" when asked to count characters. In my opinion, that's a much more interesting limitation that gets at some of the root, fundamental differences in how LLMs function.
I’m still going to have to disagree. I’d describe the idea that token granularity is the cause of letter counting limitations as a hypothesis, not as something that’s been conclusively shown as far as I’m aware.
I’m not discounting it, or even saying it’s unlikely, but its not hard to imagine other ways it could plausibly be happening.
As a side note when I mentioned training data I was not suggesting anything related to “token metadata”, or to tokens in any regard. Rather, I think it might be possible to instead improve the learning around certain types of counting in a way that could generalize.
Fine, I lied. I have a much better understanding than you. Happy?
My favorite LLM stumper is asking them to solve the farmer/wolf/chicken/grain river-crossing puzzle but with only a chicken. They usually either insist on pointless extra trips or hallucinate a wolf or grain. Liquid somehow does both and also loses track of what's where.
> The farmer can take the chicken across the river first. Then, he can go back to the original side and bring the boat back. He can then take the chicken back to the original side and leave it there. Next, he can take a bag of grain across the river. He can then go back to the original side and bring the chicken across the river. Finally, he can go back to the original side one last time and bring the chicken across the river.
One generation ended like this:
> the farmer and the chicken are both on the other side of the river, and the chicken is safe from the grain.
I find it a dumb test that proves nothing imho.
However, not very impressed with this model.
Many people can't count letters in a word?
The confusing thing about LLMs is that they speak tokens, not written text, so it's a lot more like interrogating someone who is functionally illiterate- of course they're going to be a terrible speller.
You can find much weirder things that LLMs are absurdly bad at, like "A farmer needs to get himself and a chicken across a river. His boat can hold the farmer and one piece of cargo. How does he do this?" 9 times out of 10 LLMs will pattern match this to the classic puzzle (there's usually also a sack of grain and a wolf) and start insisting on extra trips and inventing wolves. Even if a human makes the same mistake, they almost certainly would realize it after being reminded that there is no wolf, but LLMs often insist there is. o1-preview (but not -mini) seems to have cracked it, though.
The LLMs probably get it wrong because people get it wrong.
https://arbisoft.com/blogs/why-ll-ms-can-t-count-the-r-s-in-...
It doesn't.
If the input is parsed in to tokens, and the tokens split compound words, nothing about that requires "first principles" thinking to explain why LLMs struggle with getting all of the letters -- the LLM is only going down the vector path of one of the compound words...
(I don't think LLMs are sentiment or intelligent btw, I think they are giant probability machines, and the probability that the LLM will get 3 r's on a token of "berry" are very low.)
To improve character counting accuracy, LLMs may need to use more sophisticated tokenization methods, such as subword tokenization or character-level tokenization, that can preserve more information about the structure of words."
You said above that "The LLM gives you the answer it finds on the training set"
You and I both agree on that. No first principles there.
The training set -- how's it built? With tokens. We have not trained LLMs with a token structure that deals well with compound words.
If we trained LLMs with a different token structure, it is more probable that a one-shot answer for these compound word letter counting problems would be accurate.
The LLM does not need to understand what "counting is" or even "what a letter is". The LLM will regurgitate the token relationship we train it on.
Haven't found a model that does it reliably yet.
It’s certainly interesting that they can’t answer it properly, though.
...oh, you disagree with its intermediate reasoning? You think it should follow a series of logical steps that are each individually correct, rather than pulling wildly incorrect intermediate steps out of its butt and synthesizing them together in the end into a coherent answer that happens to be correct?
But that's what an LLM is! Why complain about an LLM being an (unusually transparent) LLM?
I think this example serves as a beautiful illustration of how LLMs work, and are supposed to work—even if the correspondence is inexact between (1) incorrect and irrelevant-to-us intermediate steps and (2) internal matrix multiplications that represent incorrect or invalid logic. The correspondence is inexact and probably mostly figurative, but it's still a great example of how internal nonsense can still lead to externally "correct" answers. ("Nonsense" is underselling the complex and highly sophisticated internal state that leads to "correct" answers a stunningly high percentage of the time; I'm just trying to distinguish it from formal logic or the type of reasoning that we think we do and sometimes actually do. And would do, in this example problem.)