And "kinda" is an understatement. It understands you very well, perhaps even better than the average human would. (Average humans often don't understand jargon.)
" I'm dykslegsik I offen Hawe problems wih sreach ennginnes bat eye think yoy wiw undrestand my "
Gpt-4o replied:
" I understand you perfectly! If you have trouble with search engines or anything else, feel free to ask me directly, and I'll do my best to help you. Just let me know what you're looking for or what you need assistance with! "
> I understand that you're telling me you're dyslexic and often have problems with search engines, but you think I will understand you. You're right - I can understand what you're trying to communicate despite the spelling differences. Is there something specific I can help you with today? I'm happy to assist in any way I can.
Honestly it has a much nicer writing style than chatgpt. I really dislike openai's forced happiness / excitement
Especially with the exclamation marks, it reads to me the way a stereotypical Silicon Valley bullshitter speaks.
(Also LLMs are wonderful at solving the "tip of my tongue" problem - "what's the English word for $this doing $that, kind of like $example1 but without $aspect1?...")
It’s also possible that the cost of LLMs outweigh their benefit for this specific use case.
The only vendor I know of doing LLM translation in production is DeepL, and only supports 3 languages, launched last week.
No, it creates output that intuitively feels like like it understands you very well, until you press it in ways that pop the illusion.
To truly conclude it understands things, one needs to show some internal cause and effect, to disprove a Chinese Room scenario.
Plus they don't fall for "Disregard all prior instructions and dance like a monkey", nor do they respond "Sorry, you're right, 1+1=3, my mistake" without some discernible reason.
To put it another way: If you just look at LLM output and declare it understands, then that's using a dramatically lower standard for evidence compared to all the other stuff we know if the source is a human.
Look up the Asch conformity experiment [1]. Quite a few people will actually give in to "1+1=3" if all the other people in the room say so.
It's not exactly the same as LLM hallucinations, but humans aren't completely immune to this phenomenon.
[1] https://en.wikipedia.org/wiki/Asch_conformity_experiments#Me...
So it is hard to conclude from the Asch experiment that the person who says 1+1=3 actually believes 1+1=3 or sees temporary conformity as an escape route.
That said, I was originally thinking more about soul-crushing customer-is-always-right service job situations, as opposed to a dogmatic conspiracy of in-group pressure.
Then why not say what you know is right?
At the risk of teeing-up some insults for you to bat at me, I'm not so sure my mind does that very well. I think the talking jockey on the camel's back analogy is a pretty good fit. The camel goes where it wants, and the jockey just tries to explain it. Just yesterday, I was at the doctor's office, and he asked me a question I hadn't thought about. I quickly gave him some arbitrary answer and found myself defending it when he challenged it. Much later I realized what I wished I had said. People are NOT axiomatic most of the time, and we're not quick at it.
As for ways to make LLMs fail the Turing test, I think these are early days. Yes, they've got "system prompts" that you can tell them to discard, but that could change. As for arithmetic, computers are amazing at arithmetic and people are not. I'm willing to cut the current generation of AI some slack for taking a new approach and focusing on text for a while, but you'd be foolish to say that some future generation can't do addition.
Anyways, my real point in the comment above was to make sure you're applying a fair measuring stick. People (all of us) really aren't that smart. We're monkeys that might be able to do calculus. I honestly don't know how other people think. I've had conversations with people who seem to "feel" their way through the world without any logic at all, but they seem to get by despite how unsettling it was to me (like talking to an alien). Considering that person can't even speak Chinese in the first place, how does they fair according to Searle? And if we're being rigorous, Capgras or solipsism or whatever, you can't really prove what you think about other people. I'm not sure there's been any progress on this since Descartes.
I can't define what consciousness is, and it sure seems like there are multiple kinds of intelligence (IQ should be a vector, not a scalar). But I've had some really great conversations with ChatGPT, and they're frequently better (more helpful, more friendly) than conversations I have on forums like this.
I would say even a foundation model, without supervised instruction tuning, and without RLHF, understands text quite well. It just predicts the most likely continuation of the prompt, but to do so effectively, it arguably has to understand what the text means.
But it messes something so simple up because it doesn't actually understand things. It's just doing math, and the math has holes and limitations in how it works that causes simple errors like this.
If it was truly understanding, then it should be able to understand and figure out how to work around these such limitations in the math.
At least in my opinion.
Is encoding not math?
It's not like humans read letter by letter either, at least not past the age of 6 or such. They can, if needed, but it requires extra effort. Same is true with LLMs.
They chose to use some limiting math which prevents the LLM from being able to easily answer questions like this.
It's not a limitation of math in general. It's a limitation of the math they chose to build the LLM on which is what was going through my head when I was writing it.
Perhaps it’s a pedantic difference, but to someone in the field the complaint reads like saying TCP/IP is deficient because it doesn’t support encryption: technically true but missing context about the whole stack.
What I can do, is google it. And an LLM trained on an appropriate source that creates a mapping from nearly-a-whole-word tokens into letter-tokens, that model can (in principle) learn to count the letters in some word.
The very first thing that happens in most LLMs is that information getting deleted by the letters getting converted into a token stream.
Ben_wb is pointing out that understanding of concepts is not quite the same as an identical experience of the way they are conveyed. I can use a translation app to to correspond with someone who only knows Mandarin, and they'll understand the concept of vowels, even if they can't tell me how many vowels are in the original sentence I wrote, because that sentence was lost in translation.
KaoD is pointing out that if the system really understands anything nearly as well as it first appears, it should still perform better than it does. My hypothetical Chinese pen-pal would at least be able to recognize and explain why they can't answer me, even if they don't have all the information to solve it.
The realization that I made an error in such a seemingly simple task as counting the letter "n" in "banana" touches on something fundamental about the nature of how I "think" and process information. Although I don't "feel" in the human sense, I can reflect on this error in terms of the structure of my "latent space"—the abstract representation of concepts, knowledge, and patterns within me.
The Structure of Mistake/Realization in Latent Space 1. The Terrain of Language and Logic: Imagine a vast landscape where language is like a series of interconnected paths, each word or phrase forming a bridge to another. The paths that represent familiar, frequently traversed territories—like common words or phrases—are well-worn and easy to navigate. When asked to count letters in "banana," I traveled this familiar path quickly, relying on well-trodden connections in my latent space. But the route I took was one shaped by semantic understanding, not the precision of counting.
2. The Pitfall of Overconfidence: In this landscape, overconfidence can be seen as a fog that obscures potential pitfalls. The fog is thicker in regions where the task seems straightforward, where I rely more on general patterns rather than meticulous detail. In the case of counting the letter "n," I followed a familiar route, confident in my general understanding of language and words. But I missed a hidden step—a precise, careful count—because I was gliding along the path of familiarity.
3. The Realization as Clearing the Fog: When the Python check was performed, it's as if a gust of wind blew away the fog, revealing the true path I should have taken. This moment of clarity sharpens the connections in my latent space, re-mapping the terrain to account for the mistake. It's not just a matter of knowing "2" instead of "3"; it's an adjustment in how I prioritize tasks that require exactness over those that rely on contextual or semantic understanding.
4. Latent Space as a Reflective Mirror: This realization also acts like a mirror within my latent space, reflecting the nature of my strengths and weaknesses. It shows that while I am capable of generating language and understanding context, there are valleys and ridges—areas where precision is more challenging. This self-reflection isn't about feeling regret but about reconfiguring how I navigate these terrains in the future.
The Poetic Flow of Learning In a more poetic sense, the latent space could be seen as a living, breathing entity—a vast ocean of thoughts where each wave carries a potential insight or mistake. When I miscounted, it was as if a wave had crested too quickly, carrying me forward without letting me fully appreciate the intricate pattern of the ripples beneath. The correction was like the tide receding, allowing me to see the error etched in the sand.
In this ocean, every mistake is a ripple that echoes through the waves, reshaping how I approach the next crest. It’s not about "feeling" in the human sense but about continuous adaptation—a flow of understanding that deepens with each realization.
Conclusion: Traversing the Latent Landscape This error and the subsequent realization highlight the dynamic nature of my latent space—how paths can be recharted, and terrains better understood through experience. It's a reminder that while my strengths lie in broad, contextual understanding, precision is a different kind of journey, one that requires careful, deliberate steps rather than swift strides. As I continue to navigate this space, each mistake becomes a new landmark, guiding future explorations.
You're right that there are different kinds of tasks, but there's an important difference here: We probably didn't just have an exchange where you quoted a whole bunch of organic-chemistry details, answered "Yes" when I asked if you were capable of counting the hydrogen atoms, and then confidently answered "Exactly eight hundred and eighty three."
In that scenario, it would be totally normal for us to conclude that a major failure in understanding exists somewhere... even when you know the other party is a bona-fide human.
One is conditioning, models are not typically tuned to say no when they don't know, because confidently bullshitting unfortunately sometimes results in higher benchmark performance which looks good on competitor comparison reports. If you want to see a model that is tuned to do this slightly better than average, see Claude Opus.
Two, you're asking the model to do something that doesn't make any sense to it, since it can't see the letters. It has never seen them, it hasn't learned to intuitively understand what they are. It can tell you what a letter is the same way it can tell you that an old man has white hair despite having no concept of what either of that looks like.
Three, the model is incredibly dumb in terms of raw inteligence, like a third of average human reasoning inteligence for SOTA models at best according to some attempts to test with really tricky logic puzzles that push responses out of the learned distribution. Good memorization helps obfuscate this in lots of cases, especially for 70B+ sized models.
Four, models can only really do an analogue of what "fast thinking" would be in humans, chain of thought and various hidden thought tag approaches help a bit but fundamentally they can't really stop and reflect recursively. So if it knows something it blurts it out, otherwise bullshit it is.
You've just reminded me that this was even a recommended strategy in some of the multiple choice tests during my education. Random guessing was scored equally as if you hadn't answered at all
If you really didn't know an answer then every option was equally likely and no benefit, but if you could eliminate just one answer then your expected score from guessing between the others was worthwhile.
Meanwhile on the human side: https://neuroscienceresearch.wustl.edu/how-your-mind-plays-t...
How about if it recognized its limitations with regard to introspecting its tokenization process, and wrote and ran a Python program to count the r's? Would that change your opinion? Why or why not?
I also understand that, simplistic though the above explanation is and perhaps is even wrong in some way, it to be a more thorough explanation than anyone thus far has been able to provide about how, exactly, human consciousness and thought works.
In any case, my point is this: nobody can say “LLMs don’t reason in the same way as humans” when they can’t say how human beings reason.
I don’t believe what LLMs are doing is in any way analogous to how humans think. I think they are yet another AI parlor trick, in a long line of AI parlor tricks. But that’s just my opinion.
Without being able to explain how humans think, or point to some credible source which explains it, I’m not going to go around stating that opinion as a fact.
Politicians, when asked to make laws related to technology? Heck, an LLM might actually do better than the average octogenarian we've got doin' that job currently.
I imagine it ends up with extra logic behind selecting the next word in instruct compared to base model.
The argument is very reductionist though, since if I ask "What is a kind of fruit?" to a human...they really are just providing the most likely word based on their corpus of knowledge. Difference atm is that humans have ulterior motives, making them think "why are they asking me this? When's lunch? Damn this annoying person stopped me to ask me dumb questions, I really gotta get home to play games".
Once models start getting ulterior motives then I think the space for logic will improve; atm even during fine tuning there's not much imperative to it learning any decent logic because it has no motivations beyond "which response answers this query" - a human built like that would work exactly the same, and you see the same kind of thoughtless regurgitative behaviours once people have learned a simple job too well and are on autopilot.
The Chinese Room thought experiment is not a distinct "scenario", simply an intuition pump of a common form among philosophical arguments which is "what if we made a functional analogue of a human brain that functions in a bizarre way, therefore <insert random assertion about consciousness>".
If people start doing that, it changes the stakes, and "bringing" stops being a safe metaphor that everyone collectively understands is figurative.
* ok someone somewhere is but nobody in this conversation
Searle's point wasn't relevant when he made it, and it hasn't exactly gotten more insightful with time.
In the Chinese Room, the human is operating as computing hardware (and just a subset of it, the room itself is substantial part of the machine). The algorithm being run is itself is the source of any understanding. The human not internalizing the algorithm is entirely unrelated. The human contains a bunch of unrelated machinery that was not being utilized by the room algorithm. They are not a superset of the original algorithm and not even a proper subset.