Hacker News Reader

Comment by gaganyaan | Hacker News Reader

They very much do understand words, letters, etc. Why do you think otherwise? Because of one trick question?

You are fooled into thinking that producing a set of output tokens which mimic actual human input and output tokens displays understanding. It doesn't.

"How many Rs in strawberry?" is not a trick question. That's about as straight and factual as a question could be.

>"How many Rs in strawberry?" is not a trick question.

To a system that sees letters and words, sure. To a system that doesn't, you're just asking a blind man to count how many apples are on the table and patting yourself on the back for his failure. It just doesn't make any sense to begin with.

And this is before the fact that human intelligence is rife with absurd seeming failure modes and cognitive biases.

It's frankly very telling that these token related questions are the most popular kind of questions for these discussions.

This is no testable definition of reasoning or intelligence that will cleanly separate LLMs and Humans. That is the reality today. And it should make anybody pause.

> you're just asking a blind man to count how many apples are on the table

Which input is it you think an LLM doesn’t have to be able to count the number of letters in a word you literally provide it

The text is converted to embeddings after tokenization. The neural networwk only sees vectors.

Imagine the original question is posed in English but it is translated to Chinese and then the LLM has to answer the original question based on the Chinese translation.

It's a flaw of the tokenization we choose. We can train an LLM using letters instead of tokens as the base units but that would be inefficient.

By that definition the LLM literally does not see anything. LLMs predict tokens. That's it.

It sees embeddings that is trained to encode semantic meanings.

The way we tokenize is just a design choice. Character level models(e.g. karpathy's nanoGPT) exist and are used for educational purpose. You can train it to count number of 'r' in a word.

https://x.com/karpathy/status/1816637781659254908?lang=en

The LLM sees tokens, and predicts next tokens. These tokens encode a vast world, as experienced by humans and communicated through written language. The LLM is seeing the world, but through a peephole. This is pretty neat.

The peephole will expand soon, as multimodal models come into their own, and as the models start getting mixed with robotics, allowing them to go and interact with the world more directly, instead of through the medium of human-written text.

Humans have many well-documented cognitive shortcomings and failure modes. Does that mean we don't reason either?

We aren't talking about a human mind which operates in unexpected or ways which don't conform with society. Don't bring an appeal to neurodivergence to an LLM argument.

We are talking about a computer program whose operation we understand, can observe, and can debug. Modern LLMs certainly take a lot of human effort to do this, but it can be done.

This isn't about neurodivergence. Every single human suffers from a long list of cognitive shortcomings. We just call them things like "optical illusions" and find them interesting, but don't then go on to make silly claims like "humans can't reason".

Because humans can reason. There is a difference between faulty reasoning and no reasoning at all.

Intelligence is a vast spectrum of capabilities. There are people who function normally but cant't recognize faces. There are people who can't see or understand anything on their left side. Even savantism is an example where some brains have nearly superhuman capabilities in one area, but are almost mentally retarded in other areas, like common sense, and yes even letter counting. All the letter counting failure shows is that maybe all LLMs lean a bit more towards autistic brains than towards the actual average human brain.

Ask it this.

"Please spell the word strawberry with a phonetic representation of each letter then tell me how many 'r's are in the word strawberry?"

It's gotten the answer right every time I've tried.

Another one is to just ask it to spell it out with dashes in between each letter. If anyone's wondering, it then gets it right because that changes the tokenization, changing how the model actually "sees" it.

Reply on news.ycombinator.com