Being “Confidently Wrong” is holding AI back
promptql.io
promptql.io
1. The words "the only thing" massively underplays the difficulty of this problem. It's not a small thing.
2. One of the issues I've seen with a lot of chat LLMs is their willingness to correct themselves when asked - this might seem, on the surface, to be a positive (allowing a user to steer the AI toward a more accurate or appropriate solution), but in reality it simply plays into users' biases & makes it more likely that the user will accept & approve of incorrect responses from the AI. Often, rather than "correcting" itself it merely "teaches" the AI how to be confidently wrong in an amenable & subtle manner which the individual user finds easy to accept (or more difficult to spot).
If anything, unless/until we can solve the (insurmountable) problem of AI being wrong, AI should at least be trained to be confidently & stubbornly wrong (or right). This would also likely lead to better consistency in testing.
Except they don't correct themselves when asked.
I'm sure we've all been there, many, many, many,many,many times ....
- User: "This is wrong because X"
- AI: "You're absolutely right ! Here's a production-ready fixed answer"
- User: "No, that's wrong because Y"
- AI: "I apologise for frustrating you ! Here's a robust answer that works"
- User: "You idiot, you just put X back in there"
- and so continues the vicious circle....With weak multi-turn instruction following, context data will often dominate over user instructions. Resulting in very "loopy" AI - and more sessions that are easier to restart from scratch than to "fix".
Gemini is notorious for underperforming at this, while Claude has relatively good performance. I expect that many models from lesser known providers would also have a multi-turn instruction following gap.
User: Fix this problem ...
Assistant: X
User: No, don't do X
Assistant: Y
User: No, Y is wrong too.
Assistant: X
If you reach that state, start over, and constrain your initial request to exclude X and Y. If it brings up either again, start over, and constrain your request further.
If the model is bad at handling multiple turns without getting into a loop, telling it that it is wrong is not generally going to achieve anything, but starting over with better instructions often will.
I see so many people get stuck "arguing" with a model over this, getting more and more frustrated as the model keeps repeating variations of the broken answer, without realising they're filling the context with arguments from the model for why the broken answer is right.
It makes zero difference that they've been sold as doing better if other people learn how to use them effectively and I choose to ignore how to get the best possible results out of them.
It's not hard to learn to be productive with these models.
Maybe because people expect AI systems that are touted as all-knowing, all-powerful, coming-for-your-job to be smart enough to remember what was said two turns ago?
It's not a tools fault if people insist on continuing to use them in counter-productive ways.
"If you keep pressing the back button like the IE engineers told you to, of course you will fail to go back. To go back you want to press the forward button. Are you an idiot? Press the forward button to go back, at least until the next version release when you will need to press the reload button to go back. Trust me, eventually the back button will go back, but for now only fools press the back button to go back."
Do you always insist on listening to guidance you've observed doesn't work?
It sounds immensely counter-productive.
Meanwhile I'll continue to have AI tools write the majority of my code at this point.
It is no more productive to continue to go in circles with an argumentative person who refuses to see reason.
If someone haven't learnt that lesson, they will get poor results at a whole lot more things in life than talking to AI.
If you mention X or Y, even if they're preceded by "DO NOT" in all caps, an LLM will still end up with both X and Y into its context, making it more likely it gets used.
I'm running out of ways to tell the assistant to not use mocks for tests, it really really wants to use them.
(And yes, it's a horrible workaround)
I think often it's not required to completely start over: just identify the part where it goes off the rails, and modify your prompt just before that point. But yeah, basically the same process.
They tend to very quickly lose useful context of the original problem and stated goals.
Is there anything else I can help you with?
> Yeah I think our jobs are safe.
> Why doesn’t anyone acknowledge loops like this?
Now what you've got is a junk-soup where the original problem is buried somewhere in the pile.
Best approach I've found is to start a fresh conversation with the original problem statement and any improvements/negative reinforcements you've gotten out of the LLM tacked on.
I typically have ChatGPT 5 Thinking, Claude 4.1 Opus, Grok 4, and Gemini 2.5 Pro all churning on the same question at once and then copy-pasting relevant improvements across each.
That means that positively worded instructions ("do x") work better than negative ones ("don't do y"). The more concepts that you don't want it to use / consider show up in the context, the more they do still tend to pull the response towards them even with explicit negation/'avoid' instructions.
I think this is why clearing all the crap from the context save for perhaps a summarizing negative instruction does help a lot.
> positively worded instructions ("do x") work better than negative ones ("don't do y")
I saw someone on Twitter put it eloquently: something about how, just like little kids, the moment you say "DON'T DO XYZ" all they can think about is "XYZ..."
In teacher school, we're told to always give kids affirmative instructions, ie "walk" instead of "don't run". The idea is that it takes more energy for a child to figure out what to do.
While I agree, and also use your work around, I think it stands to reason this shouldn't be a problem. The context had the original problem statement along with several examples of what not to do and yet it keeps repeating those very things instead of coming up with a different solution. No human would keep trying one of the solutions included in the context that are marked as not valid.
Exactly. And certainly not a genius human with the memory of an elephant and a PhD in Physics .... which is what we're constantly told LLMs are. ;-)
In theory you should be able to get a multiplicative effect on context window size by consolidating context into it's most distilled form.
30,000 tokens of wheel spinning to get the model back on track consolidated to 500 tokens of "We tried A, and it didn't work because XYZ, so avoid A" and kept in recent context
> No human would keep trying one of the solutions included in the context that are marked as not valid.
This is going to age like "full self driving cars in 5 years". Yeah it'll gain capabilities, maybe it does do 80% of the work, but it still can't really drive itself, so it ultimately won't replace you like people are predicting. The money train assures that AGI/FSD will always be 6-18 months away, despite no clear path to solving glaring, perennial problems like the article points out.
I vividly remember when some folks from Microsoft come to my school to give a talk at some Computer Science event and proclaimed that yep, we have working AGI, the only limiting factor is hardware, but that should be resolved in about ten years.
This was in 2001.
Some grifts in technology are eternal.
How long before there's an AI smart enough to say 'no' to half the terrible ideas I'm assigned?
My impression is rather: there exist two kinds of people who are "very invested in this illusion":
1. People who want to get rich by either investing in or working on AI-adjacent topics. They of course have an interest to uphold this illusion of magic.
2. People who have a leftist agenda ("we will soon all be replaced by AI, so politics has to implement [leftist policy measures like UBI]"). If people realize that AI is not so powerful, after all, such leftist political measures whose urgency was argued with the (hypothetical) huge societal changes that will be caused by AI will not have a lot backing in society, or at least not considered to be urgently implemented by society.
The more leftist position ever since the days of Marx has been that "right rather than being equal would have to be unqueal" to be equitable given that people have different needs, to paraphrase from Critique of the Gotha Program - UBI is in direct contradiction to socialist ideals of fairness.
The people I see pushing UBI, on the contrary, usually seems motivated either by the classically liberal position of using it to minimise the state, or driven by a fear of threats to the stability of capitalism. Saving capitalism from perceived threats to itself isn't a particularly leftist position.
It doesn't mean these loops aren't an issue, because they are, but once you stop engaging with them and cut them off, they're a nuisance rather than a showstopper.
"So what if you have to throw out a week's worth of work. That's how these things work. Accept it and you'll be happier. I have and I'm happy. Don't you see that it's OK to have your tool corrupt your work half way through. It's the future of work and you're being left behind by not letting your tools corrupt your work arbitrarily. Just start over like a real man."
But I've had it consistently happens to me on tiny contexts (e.g. I've had to spend time trying - and failing - to get it to fix a mess it was making with a straightforward 200-ish line bash script).
And its also very frequently happened to me when I've been very careful with my prompts (e.g. explicitly telling it to use a specific version of a specific library ... and it goes and ignores me completely and picks some random library).
I would be willing to record myself using them across paid models with custom instructions and see if the output is still garbage.
That's not useful.
You case is no different from:
- AI: "The capital of France is Paris"
- User: "This is wrong, it changed to Montreal in 2005"
- AI: "You're absolutely right! The capital of France is Montreal"
Nope—Paris is the capital of France and has been for centuries. Montreal is in Quebec, Canada. France’s presidency (Élysée), parliament (Assemblée nationale and Sénat), and ministries are all in Paris.I was reminded of this this morning when using Claude code (which I love) and I was confidently incorrect about a feature of my app. Claude proposed a plan, I said “great, but don’t build part 3, just use the existing ModuletExist”. Claude tied itself in knots because it believes me.
(The module does exist in another project I’m working on)
Probably the ideal would be to have a UI / non-chat-based mechanism for discarding select context.
Yes! I often find myself overthinking my phrasing to the nth degree because I've learned that even a sprinkle of bias can often make the LLM run in that direction even if it's not the correct answer.
It often feels a bit like interacting with a deeply unstable and insecure people pleasing person. I can't say anything that could possibly be interpreted as a disagreement because they'll immediately flip the script, I can't mention that I like pizza before asking them what their favorite food is because they'll just mirror me.
Exactly. One could argue that this is just an artifact from the fundamental technique being used: it’s a really fancy autocomplete based on a huge context window.
People still think there’s actual intelligence in there, while the actual problems by making these systems appear intelligent is mostly algorithms and software managing exactly what goes into these context windows at what place.
Don’t get me wrong: it feels like magic. But I would argue that the only way to recognize a model being “confidently wrong” is to let another model, trained on completely different datasets with different techniques, judge them. And then preferably multiple.
(This is actually a feature of an MCP tool I use, “consensus” from zen-mcp-server, which enables you to query multiple different models to reach a consensus on a certain problem / solution).
Humans have meta-cognition that helps them judge if they're doing a thing with lots of assumptions vs doing something that's blessed.
Humans decouple planning from execution right? Not fully but we choose when to separate it and when to not.
If we had enough data on here's a good plan given user context and here's a bad plan, it doesn't seem unreasonable to have a pretty reliable meta cognition capability on the goodness of a plan.
* there are already lots of "reasoning" models trying meta-cognition, while still getting simple things wrong
or:
* the models aren't doing cognition, so meta-cognition seems very far away
Could real-time observability into the network's internals somehow feed back into the model to reduce these hallucination-inducing shortcuts? Like train the system to detect when a shortcut is being used, then do something about it?
But it happened at a time where hype can be delivered at a magnitude never before seen by humanity as well to a degree of volume that is completely unnatural by any standard set previously by hype machines created by humanity. Not even landing on the moon has inundated people with as much hype. But inevitably like landing on the moon, humanity is suffering from hype fatigue.
Too much hype makes us numb to the reality of how insane the technology is.
Like when someone says the only thing stopping LLMs is hallucinations… that is literally the last gap. LLMs cover creativity, comprehension, analysis, knowledge and much more. Hallucinations is it. The final problem is targeted and boxed into something much more narrower then just build a human level AI from scratch.
Don’t get me wrong. Hallucinations are hard. But this being the last thing left is not an underplay. Yes it’s a massive issue but yes it is also a massive achievement to reduce all of agi to simply solving just an hallucination problem.
What you are missing here is that the "hallucinations" you don't like and the "results" you do like are, in terms of the underlying process, exactly the same thing. They are not an aberration you can remove. Producing these kinds of results without "hallucinations" is going to require fundamentally different techniques. It's not a "last gap".
Humans have a condition called schizophrenia where we literally are incapable of differentiating hallucination and reality. What that capability is, is something we need to find out and discover for both ourselves and LLMs.
For example: Mathematically speaking it's possible to know how far away an inferenced point is away from a cluster of real world data. That delta when fed back into the neural network can allow the LLM to know how speculative a response is. From there we can feed the response back into itself for refinement.
And even if we were to cure schizophrenia in humans, just what makes you think that it would apply to LLMs? Having an extremely weak conceptual model of the world and not being able to reason out of rather simple problems (like LLMs struggle with) isn't schizophrenia.
This oversimplified explanation which posits that neural networks are just like human brains has truly gone too far now.
> Mathematically speaking it's possible to know how far away an inferenced point is away from a cluster of real world data.
And mathematically speaking, how would you accomplish this? As you probably know LLMs don't operate on conceptual ideas, they operate on tokens. That's why LLMs tend to fail when asked to do things that aren't well represented in their training data, they don't have a working model of the world even if they can fake it to a certain degree.
A weak conceptual model of the world is the problem. But realize humans also have a weak conceptual model of the world as well and make a bunch of hallucinations based on that weak model. For example many people are still making the claim about LLMs that it’s all stochastic parroting when it’s been proven that it’s not. That is an hallucination. Or the people betting (and not) on the financial success of crypto or AI. We don’t know how either of these things will pan out but people on either team act as if they know definitively. A huge part of human behavior is driven by hallucinations that fill in gaps.
> And mathematically speaking, how would you accomplish this? As you probably know LLMs don't operate on conceptual ideas, they operate on tokens. That's why LLMs tend to fail when asked to do things that aren't well represented in their training data, they don't have a working model of the world even if they can fake it to a certain degree.
It’s not an incorrect model of the world as technically both you and an LLM ultimately have an incorrect model of the world and both you and the LLM fake it. The best you can say is that the LLM has a less accurate approximation of the world than you but ultimately both you and the LLM hold an incorrect model and both you and the LLM regularly hallucinate off of it. You also make up bullshit on things not well represented in your own model.
But like I said we are often (and often not) aware of our own bullshit so providing that to the LLM quantitatively will help it too.
The LLM is not just trained on random tokens it’s trained on highly specific groups of tokens and those groups of represent conceptual ideas. So an LLM is 100 percent trained on concepts and tokens are only an encoding of that concept.
If a group of tokens represents a vector then we can for sure calculate distance between vectors. We know that there are also different types of vectors represented at each layer of the feed forward network that encode reasoning and not just the syntactic order of the tokens.
Like literally there is not very much training data of a human giving instructions to someone to write code and the associated code diff. The fact that an LLM can do this to a useable degree without volumes of similar training data speaks to the fact it knows concepts. This is the same tired argument that has been proven wrong. We already know LLMs aren’t just parroting training data as the majority of the agentic coding operations we currently use LLMs for actually don’t have associated training data to copy.
Given that we know all of these embeddings from the training data (the model had to calculate the embeddings at one point) we can encode proximity and distance into the model via addition and subtraction of the magnitude of vectors and from this we extract a number that ascertains distance between vectors embeddings.
Imagine a best fit 2D curve through a scatter plot of data points. But at the same time that curve has a gradient color along it. Red indicates its very close to existing data points blue indicates its far. We can definitely derive and algorithm that calculates the additional “self awareness” dimension here encoded in color and this can extend to the higher dimensional encoding that is the LLM.
If an LLm is aware that the output is red or blue then it can sort of tell that if the line is blue it’s likely to be an hallucination.
I should've said that the model is "missing", not "weak" when talking about LLMs, that was my mistake. Yes I'm a human with an imperfect and in many aspects incorrect conceptual model of the world, that is true. The following aren't real examples, they're hyperbolic to better illustrate the category of errors I'm talking about.
If someone asks me "can I stare into the sun without eye protection", my answer isn't going to change based on how the question is phrased because I conceptually understand that the radiation coming from the sun (and more broadly, intense visible radiation emitted from any source) causes irreversible damage to your eyes, which is a fact stored in my conceptual understanding of the world.
However LLMs will flip flop based on tone and phrasing of your question. Asked normally, they will warn you about the dangers of staring into the sun, but if your question hints at disbelief, they might reply "No you're right, staring into the sun isn't that bad".
I also know that mirrors reflect light, which allows me to intuitively understand that staring at the sun through a mirror is dangerous without being explicitly taught that fact.
If you ask an LLM whether staring into a mirror which is pointed at the sun (oriented such that you see the sun through the mirror) is safe, they might agree that it's safe to do so, even though they "know" that staring into the sun is dangerous, and they "know" that mirrors reflect light. Presumably this is because their training data doesn't explicitly state that staring at a mirror is dangerous.
The way the question is framed can completely change their answer which betrays their lack of conceptual understanding. Those are distinctly different problems. You might say that humans do this too, but we don't call that intelligent behavior, and we tend to have a low opinion of those who exhibit this behavior often.
Like if I told it solve a complex puzzle equation not in its training data and it correctly solved that problem. We know from the low probability of arriving at that solution from random chance that the LLM must know and understand and reason to arrive at that solution.
Now you’re saying you perturb the input with some grammar changes but leave everything else the same and the LLM will now produce a wrong answer. But this doesn’t change the fact that it was able to get the right answer.
Humans can be dumb and inconsistent. LLMs can be dumb and inconsistent too. This happens to be a quirk of the LLM. But you cannot deny that it is intelligent on the sole fact that LLMs can produce output that we know for sure can only be arrived at through reasoning.
Having a conceptual understanding means that you always provide the same answer to a conceptually equivalent question. Producing the wrong answer when a question is rephrased is indicative of rote memorization.
The fact that it provided the right answer at one point is only indicative of memorization, not understanding which is precisely the difference between sometimes getting it right and always getting it right.
False. I can lie right? I can shift. I don't need to be consistent. And I don't need to consistently understand something. I can understand something right now and suddenly not understand later. This FITS the definition of understanding a concept.
But If I gave an answer that has such a low probability of being correct, and the answer is correct, then the answer arrived at by random chance. If the answer wasn't arrived at by random chance it must be reasoning AND understanding.
The logic is inescapable.
Not any definition that I would agree with, that's for sure.
When I say I can understand something now and then not understand something later it doesn’t violate the definition of the word. Now you are making a claim that your personal definition of understanding is violated but that’s also a lie. It’s highly unlikely.
First of all death. I understand something now. Then I die, I don’t understand something later due to loss of consciousness.
Amnesia. I understand something now and I don’t understand something later due to loss of memory.
In both cases someone understood something now and didn’t later. Every human understands this conceptually. Don’t lie to my face and say you don’t agree with the definition. This is fundamental.
The act of understanding something now and then not understanding something later exists as not only some virtual construct by human language but it exists in REALITY.
What happened here is that when I pointed out the nuances of the logic you were too stubborn to reformulate your conclusion. It’s typical human behavior. Instead you are unconsciously re-scaffolding the rationale in order to fit your pre existing idea.
If you’re capable of thinking deeper you’ll be able to see what I’m in essence talking about this:
In the gap between prompt and response. The LLM is capable of understanding the prompt and capable of reasoning about the prompt. It does so on an ephemeral and momentary basis. We can’t control when it will do it and that’s the major issue. But it does do it often enough that we know the LLM has reasoning capabilities however rudimentary and inconsistent because the answer it arrives at via the prompt is too low probability to be arrived at using any other means OTHER than reasoning.
What we got instead is a bunch of wisecracking programmers who like to remind everyone of the 90–90 rule, or the last 10 percent.
AGI is part of the last 10 percent rule. But like that's the entire issue. 90% is still 90% progress. That is massive. And the hype surrounding LLMs has made people forget how far 90% is. People are going, "LLMs are retarded because it has the IQ of a 5 year old". They don't realize how even getting it to the level of a 5 year old was impossible for decades and decades.
And then you mouth off a buzz phrase not even coined by a programmer but repeated to the point of annoyance about how the final 10 percent is always the hardest as if programmers who copy the phrase are so smart.
Bro the last 10 percent being the hardest doesn’t mean the previous 90 percent didn’t happen. The first 90 percent is a feat in itself and LLMs can now even do PRs. That was a feat no one just 5 years ago could have predicted was possible in our lifetimes.
Idiot programmers and their generic wise cracks were the ones saying that AI would never be able to pass the Turing test and this was just 4 years ago.
> To accomplish X you can just use Y!
But Y isn't applicable in this scenario.
> Oh, you're absolutely right! Instead of Y you can do Z.
Are you sure? I don't think Z accomplishes X.
> On second thought you're absolutely correct. Y or Z will clearly not accomplish X, but let's try Q....
It's not obvious how long until that point or what form it will finally take, but it should be obvious that it's going to happen at some point.
My speculation is that until AI starts having senses like sight, hearing, touch and the ability to learn from experience, it will always be just a tool/help/aider to someone doing a job, but could not possibly replace that person in that job as it lacks the essential feedback mechanisms for successfully doing that job in the first place.
Like you realize humans hallucinate too right? And that there are humans that have a disease that makes them hallucinate constantly.
Hallucinations don’t preclude humans from being “intelligent”. It also doesn’t preclude the LLM from being intelligent.
Minority != wrong, with many historic examples that imploded in spectacular fashion. People at the forefront of building these things aren't immune from grandiose beliefs, many of them are practically predisposed to them. They also have a vested interest in perpetuating the hype to secure their generational wealth.
The ai can easily answer correctly complex questions NOT in its data set. If it is generating answers to questions like these out of thin air which fits our colloquial definition of intelligence.
"Is X true" -> "Yes, X is true."
"Is X a myth?" -> "Yes, X is a myth"
"Is Y a myth?" (where X = Y, rephrased) -> "No, Y is true"
Even when they're provided with all the facts required to reach the correct answer through simple reasoning, they'll often fail to do so.
Worse still, sometimes they can be told what the correct answer is, with a detailed step-by-step explanation, but they'll still refuse to accept it as true, continuing to make arguments which were debunked by the step-by-step explanation.
All state of the art models exhibit this behavior, and this behavior is inconsistent with any definition of intelligence.
They do this really impressive stuff like generate code and hold conversations that makes them seem intelligent, but then they fail at these extremely basic tasks which, to me, proves that it's all an illusion.
It doesn't understand the instructions you give it, it doesn't even understand the answer it gives you. It just consumes and generates tokens. Sure it works pretty well and it's pretty cool stuff, but it's not AI.
The fact of the matter is that as retarded and as stupid as the LLM is the fact that it’s so prevalent in the world today is because it gets answers right. We ask it things not in its training data and it produces an answer out of a range of possibilities that is to low probability to be produced by ANY other thing other than actual reasoning and logic.
You need to see nuance here and make your assessment of LLMs NOT based on singular aspects of facts. LLMs get shit wrong all the time they also get shit right all the time and so do humans. What does that look like holistically?
Look at the shit it’s getting right . If it’s getting stuff right that’s not in the training data then some mechanism in there is doing actual “thinking” and when it gets shit wrong well, you get shit wrong too. All getting shit wrong does to you is make you a dumbass it doesn’t make you not an intelligent entity. You don’t lose that status as soon as you do something incredibly stupid which I’m sure you’ve done often enough in your life to know the difference.
They get simple, rephrased, but conceptually equivalent questions really wrong and they do this:
1. while the context already contains their previous answer to the original question (which was correct),
2. while the context contains all background information on the topic that would allow an intelligent being to arrive at the correct answer through simple logical deduction,
3. without recognizing or acknowledging that they provided a conflicting answer (without being prompted),
4. while denying that the two answers are contradictory if that fact is pointed out to them,
5. while fabricating a list of bogus reason justifying a different answer if pressed for an explanation.
That's one common failure mode, the other common failure mode is where they uncritically accept our own erroneous corrections even when the correction contain obviously flawed reasoning.
This behavior demonstrates a fundamental lack of conceptual understanding of the world and points at rote memorization in the general case. Maybe LLMs develop a more conceptual understanding of a certain topic when they've been benchmaxxed on that topic? I don't know, I'm not necessarily arguing against that, not today anyway.
But these errors are a daily occurrence in the general case when it comes to any topic they haven't been benchmaxxed for - they certainly don't have a conceptual understanding of cooking, baking, plumbing, heating, electrical circuits, etc.
We don't know how or why it gets things wrong. The LLMs are a black box. There are infinite ways it can get something wrong, so you cannot base your reasoning off of this simply because you don't know HOW it got things wrong. It may be similar to the way humans get things wrong or it may be different.
>This behavior demonstrates a fundamental lack of conceptual understanding of the world and points at rote memorization in the general case. Maybe LLMs develop a more conceptual understanding of a certain topic when they've been benchmaxxed on that topic? I don't know, I'm not necessarily arguing against that, not today anyway.
False. The LLM could be lying right? We don't know if these things are lying or if they lack actual understanding.
>But these errors are a daily occurrence in the general case when it comes to any topic they haven't been benchmaxxed for - they certainly don't have a conceptual understanding of cooking, baking, plumbing, heating, electrical circuits, etc.
You're failing to look at the success modes. Unlike the failure modes, if it succeeds in answering a prompt for which NO TRAINING data exists we know for a fact it used reasoning and it understood what it was being asked. We don't know what happened if it's a failure BUT we do know understanding and reasoning occured if it was NOT a failure mode ON a prompt with zero training data.
How?
Because of probability. There are two possible ways to get an answer correct. Random chance. Or reasoning. We know the number of incorrect answers far out number the number of correct answers.
Therefore from logic we know that LLMs MUST use reasoning and understanding to arrive at a correct answer. The logic follows from probability.
Now this does not mean the LLM does not lie, it does not mean that the LLM is consistently understanding a concept, it does not give it the same conceptual style of thinking that a human does.
But we do know that journey from prompt A to response B on a prompt and response pair that did not exist in training data, we know that reasoning and understanding happened in this gap. This fits our colloquial logical understanding of the world, of probability, and of the definition of the words reasoning and understanding.
The issue we face now is how do we replicate that gap consistently.
They also dont have an internal world model. Well I don't think so, but the debate is far from settled. "Experts" like the cofounders of various AI companies (whose livelihood depends on selling these things) seem to believe that. Others do not.
So presumably we have a solid, generally-agreed-upon definition on intelligence now?
> autocompleting things with humanity changing intelligent content.
What does this even mean?
Because we can do this, by logic a universally agreed upon definition exists. Otherwise we wouldn’t be able to do this.
Of course the boundaries between what’s not intelligent and what is, is where things are not as universally agreed upon. Which is what you’re referring to and unlike you I am charitably addressing that nuance rather then saying some surface level bs.
The thing is the people who say the LLM (which obviously exists at this fuzzy categorical boundary) is not intelligent will have logical paradoxes and inconsistencies when they examine there own logic.
The whole thing is actually a vocabulary problem as this boundary line is an arbitrary definition given to a made up word that humans created. But one can still say an LLM is well placed in the category of intelligent not by some majority vote but because that placement is the only one that maintains logical consistency with OTHER entities or things all humans place in the intelligent bucket.
For example a lot of people in this thread say intelligence requires actual real time learning, therefore an LLM is NOT intelligent. But then there are humans who literally have anterograde amnesia and they literally cannnot learn. Are they not intelligent? Things like this are inconsistent and it happens frequently when you place LLMs in the not intelligent bucket.
State your reasoning for why your stance is "not intelligent" and I can point out where the inconsistencies lie.
Go check out anthropic's careers page and see just how few positions even require a formal training in statistics.
Meanwhile I don't see a lot of real statisticians who are that hyped about LLMs. More importantly, it feels like there aren't even that many scientists at the AI companies.
Your average programmer does not have nearly the "question your assumptions and test your beliefs" training that an actual scientist has, which is funny since nearly every bug in code is caused by an assumption you shouldn't have made and should have tested.
You're shocked because you hallucinated an assumption of something I never claimed.
Hallucinations? Does that sound similar to something?
A developer that hallucinates at work to the extent that LLMs does would probably have issues getting their PRs past code reviews a lot.
Because of this we should euthanize all schizophrenics. Just stab them to death or put a bullet in their heads right? I mean they aren’t intelligent or sentient so you shouldn’t feel anything when you do this.
I’m baffled as to why people think of this in terms of PRs. Like the LLM is intelligent but everyone’s like oh it’s not following my command perfectly therefore it’s not intelligent.
There are cases where humans lose all ability to form long term memories and outside of a timed context window they remember nothing. That context window is minutes at best.
According to your logic these people have no actual intelligence or sentience. Therefore they should be euthanized. You personally can grab a gun and execute each of these people one by one with a bullet straight to the head because clearly these people have no actual intelligence or sentience. That’s the implication of your logic.
https://en.m.wikipedia.org/wiki/Anterograde_amnesia
It’s called anterograde amnesia. Do you see how your logic can justify gassing all these people holocaust style?
When I point out the flaw in your logic do you use the new facts to form a new conclusion? Or do you rearrange the facts to maintain support for your existing conclusion?
If you did the later I hate to tell you this, it wasn’t very intelligent. It was biased. But given that you’re human, that’s what you most likely did and it’s normal. But pause for a second and try to do the former of using the new facts to form a different more nuanced conclusion.
Pronoun and noun wordplay aside ( 'Their' ... `themselves` ) I also agree that LLMs can correct the path being taken, regenerate better, etc...
But the idea that 'AI' needs to be _stubbornly_ wrong ( more human in the worst way ) is a bad idea. There is a fundamental showing, and it is being missed.
What is the context reality? Where is this prompt/response taking place? Almost guaranteed to be going on in a context which is itself violated or broken; such as with `Open Web UI` in a conservative example: Who even cares if we get the responses right? Now we have 'right' responses in a cul-de-sac universe. This might be worthwhile using `Ollama` in `Zed` for example, but for what purpose? An agentic process that is going to be audited anyway, because we always need to understand the code? And if we are talking about decision-making processes in a corporate system strategy... now we are fully down the rabbit hole. The corporate context itself is coming or going on whether it is right/wrong, good/evil, etc... as the entire point of what is going on there. The entire world is already beating that corporation to death or not, or it is beating the world to death or not... so the 'AI' aspect is more of an accelerant of an underlying dynamic, and if we stand back... what corporation is not already stubbornly wrong, on average?
How is that wordplay? Those are the correct pronouns.
It’s like saying you built a 3D scene on a 2D plane. You can employ clever tricks to make 2D look 3D at the right angle, buts it’s fundamentally not 3D, which obviously shows when you take the 2D thing and turn it.
It seems like the effectiveness plateau of these hacks will soon be (has been?) reached and the smoke and mirrors snake oil sales booths cluttering Main Street will start to go away. Still a useful piece of tech, just, not for every-fucking-thing.
The author proposes ways for an AI to signal when it is wrong and to learn from its mistakes. But that mechanism feeds back to the core next token matcher. Isn't this just replicating the problem with extra steps?
I feel like this is a framing problem. It's not that an LLM is mostly correct and just sometimes confabulates or is "confidently wrong". It's that an LLM is confabulating all the time, and all the techniques thrown at it do is increase the measured incidence of LLM confabulations matching expected benchmark answers.
Technically, I can't prove that they're wrong, novel solutions sometimes happen, and I guess the calculus is that it's likely enough to justify a trillion dollars down the hole.
His big idea is that evolution/advancements don't happen incrementally, but rather in unpredictable large leaps.
He wrote a whole book about it that's pretty solid IMO: "Why Greatness Cannot Be Planned: The Myth of the Objective."
[0] https://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_t... [1] https://en.wikipedia.org/wiki/HyperNEAT
I guess the bitter lesson is gospel now, which doesn't sit right with me now that we're past the stage of Moore's Law being relevant, but I'm not the one with a trillion dollars, so I don't matter.
I remember a few years ago, we were planing to make some kind of math forum for students in the first year of the university. My opinion was that it was too easy to do it wrong. On one way you can be like Math Overflow were all the questions are too technical (for first year of the university) and all the answers are too technical (first year of the university). On the other way, you can be like Yahoo! Answers, where more than half of the answers were "I don't know", with many "I don't know" per question.
For the AI, you want to give it some room to generalize/bullshit. It one page says that "X was a few months before Z" and another page says that "Y was a few days before Z", than you want an hallucinated reply that says that "X happened before Y".
On the other hand, you want the AI to say "I don't know.". They just gave too little weight to the questions that are still open. Do you know a good forum where people post questions that are still open?
Totally! In my mind I’ve been playing with the phrase: it’s good at _fuzzy_ things. For example IMO voice synthesis before and after this wave of AI hype is actually night and day! In part, to my fuzzy idea, because voice synthesis isn’t factual, it’s billions of little data points coming together to emulate sound waves, which is incredibly fuzzy. Versus code, which is pointy: it has one/few correct forms, and infinite/many incorrect forms.
I predict we'll get a few research breakthroughs in the next few years that will make articles like this seem ridiculous.
Re training data - We have synthetic data, and we probably haven't hit a wall. Gpt-5 was only 3.5 months after o3. People are reading too much into the tea leaves here. We don't have visibility into the cost of Gpt-5 relative to o3. If it's 20% cheaper, that's the opposite of a wall, that's exponential like improvement. We don't have visibility into the IMO/IOI medal winning models. All I see are people curve fitting onto very limited information.
A "frozen mind" feels like something not unlike a book - useful, but only with a smart enough "human user", and even so be progressively less useful as time passes.
>Doesn't seem like a problem that needs to be solved on the critical path to AGI.
It definitely is one. I know we are running into definitions, but being able to form novel behavior patterns based on experience is pretty much the essence of what intelligence is. That doesn't necessary mean that a "frozen mind" will be useless, but it would certainly not qualify as AGI.
>We don't have visibility into the IMO/IOI medal winning models.
There are lies, damn lies and LLM benchmarks. IMO/IOI is not necessarily indicative of any useful tasks.
But every time you tried to get him to do something you'd have to teach him from first principles. Good luck getting ChatStein to interact with the internet, to write code or design a modern airplane. Even in physics, he'd be using antiquated methods and assumptions, this getting worse as time progresses(like sib comment I believe was alluding to).
And don't even get me started on the language barrier.
I recently read this short story[1] on the topic so it's fresh on my mind.
I‘ve yet to see a convincing article for artificial training data.
This. Lack of any way to incorporate previous experience seems like the main problem. Humans are often confidently wrong as well - and avoiding being confidently wrong is actually something one must learn rather than an innate capability. But humans wouldn't repeat same mistake indefinitely.
The feedback you get is incredibly entangled, and disentangling it to get at the signals that would be beneficial for training is nowhere near a solved task.
Even OpenAI has managed to fuck up there - by accidentally training 4o to be a fully bootlickmaxxed synthetic sycophant. Then they struggled to fix that for a while, and only made good progress at that with GPT-5.
But I agree that being confidently wrong is not the only thing they can't do. Programming, great, maths, apparently great nowadays, since Google and OpenAI have something that could solve most problems on the IMO, even if the models we get to see probably aren't models that can do this, but LLMs produce crazy output when asked to produce stories, they produce crazy output when given too long confusing contexts and have some other problems of that sort.
I think much of it is solvable. I certainly have ideas about how it can be done.
I think the next iteration of LLM is going to be "interesting", i.e. now that all the websites they used to freely scrape have been increasingly putting up walls.
Except nvidia perhaps
Uber successfully turned a war chest into a partial monopoly of all ride hailing for significant chunks of the world. That was the clear plan from the start, and was always meant to own the market so they could extract whatever rent they want.
Amazon reinvested heavily while competition floundered in order to literally own the market, and has spent every second since squeezing the market, their partners, everyone in the chain to extract ever more money from it.
None of those are even close to buying absurdly overpriced hardware from a monopoly and reselling access to that hardware for less than it costs to run and doing huge PR sweeps about how what you are building might kill everyone so we should obviously give them trillions in government dollars because if an American company isn't the one to kill everyone than we have failed.
"Not terribly knowledgeable in the field"?
You’re right in that it’s obviously not the only problem.
But without solving this seems like no matter how good the models get it’ll never be enough.
Or, yes, the biggest research breakthrough we need is reliable calibrated confidence. And that’ll allow existing models as they are to become spectacularly more useful.
Ha, that almost seems like an oxymoron. The previous encounters can be the new training data!
What would be the point of training an LLM on bot answers to human questions? This is only useful if you want to get an LLM that behaves like an already existing LLm
But memory is a minor thing. Talking to a knowledgeable librarian or professor you never met is the level we essentially need to get it to for this stuff to take off.
And now, in some cases for a while, it is training on its own slop.
They are at their most useful when it is cheaper to verify their output than it is to generate it yourself. That’s why code is rather ok; you can run it. But once validation becomes more expensive than doing it yourself, be it code or otherwise, their usefulness drops off significantly.
If what I am doing is summarizing data and it will likely have uncertainty as a result, I can include statistics in the specification of what I want.
I have also been impressed from time to time where Claude Code catches a mistake I would have written. For example, I asked it to create a configuration file with some names of my staff to use for a query. It then ran the query and noticed that one name I gave was not in the database, but that there was a similar name, and it recommended changing the config.
I am pessimistic about whether these tools are intelligent or will ever achieve intelligence, but where they are useful, we should use them.
It's the difference between a categorical distribution and a Dirichlet. https://en.wikipedia.org/wiki/Dirichlet_distribution
The softmax probabilities are usually not a very good indication of uncertainty, as the model is often overconfident due to neural collapse. The uncertainty in the softmax probabilities is a good indication though, and can be used to detect out-of-distribution entries or poor predictions.
Isn't it obvious?
It's all built around probability and statistics.
This is not how you reach definitive answers. Maybe the results make sense and maybe they're just nice sounding BS. You guess which one is the case.
The real catch --- if you know enough to spot the BS, you probably didn't need to ask the question in the first place.
Yes, the world is probabilistic.
> This is not how you reach definitive answers.
Do go on? This is the only way to build anything approximating certainty in our world. Do you think that ... answers just exist? What type of weird deterministic video game world do you live in where this is not the case?
I'm certain this simple question has a definitive answer.
AI: “I’ve deployed the API data into your app, following best practices and efficient code.”
Me: “Nope thats totally wrong and in fact you just wrote the API credential into my code, in plaintext, into the JavaScript which basically guarantees that we’re gonna get hacked.”
AI: “You’re absolutely right. Putting API credentials into the source code for the page is not a best practice, let me fix that for you.”
“LLMs don’t know what they don’t know” https://blog.scottlogic.com/2025/03/06/llms-dont-know-what-t...
But I wouldn’t say it is the only problem with this technology! Rather, it is a subtle issue that most users don’t understand
As Mazer Rackham from Ender's Game said: "Only the enemy shows you where you are weak."
It makes you a walking database --- an example of savant syndrome.
Combine this with failure on simple logical and cognitive tests and the diagnosis would be --- idiot savant.
This is the best available diagnosis of an LLM. It excels at recall and text generation but fails in many (if not most) other cognitive areas.
But that's ok, let's use it to replace our human workers and see what happens. Only an idiot would expect this to go well.
https://nypost.com/2024/06/17/business/mcdonalds-to-end-ai-d...
LLMs don't do well at following style instructions, and existing memory systems aren't adequate for "remembering" my style preferences.
When you ask for one change, you often get loads of other changes alongside it. Transformers suck at targeted edits.
The hallucination problem and the sycophancy/suggestibility problem (which perhaps both play into the phenomenon of being "confidently wrong") are both real and serious. But they hardly form a singular bottleneck for the usefulness of LLMs.
The key feature of formalization is the ability to create statements, and test statements for correctness. ie, we went from fuzzy feel-good thinking to precise thinking thanks to the formalization.
Furthermore, the ingenuity of humans is to create new worlds and formalize them, ie we have some resonance with the cosmos so to speak, and the only resonance that the LLMs have is with their training datasets.
It's literally just a statistical model that guesses what you want based on the prompt and a whole bunch of training data.
If we want a black box that's AGI/SGI, we need a completely new paradigm. Or we apply a bunch of old-school AI techniques (aka. expert systems) to augment LLMs and get something immediately useful, yet slightly limited.
RIght now LLMs do things and are somewhat useful. Short of some expectations, butter than others, but yeah, a statistical model was never going to be more than the sum of its training data.
Yesterday I asked ChatGPT a really simple, factual question. "Where is this feature on this software?" And it made up a menu that didn't exist. I told "No,, you're hallucinating, search the internet for the correct answer" and it directly responded (without the time delay and introspection bubbles that indicate an internet search) "That is not a hallucination, that is factually correct". God damn.
As the most well known example: Anthropic examined their AIs and found that they have a "name recognition" pathway - i.e. when asked about biographic facts, the AI will respond with "I don't know" if "name recognition" has failed.
This pathway is present even in base models, but only results in consistent "I don't know" if AI was trained for reduced hallucinations.
AIs are also capable of recognizing their own uncertainity. If you have an AI-generated list of historic facts that includes hallucinated ones, you can feed that list back to the same AI and ask it about how certain it is about every fact listed. Hallucinated entries will consistently have less certainty. This latent "recognize uncertainty" capability can, once again, be used in anti-hallucination training.
Those anti-hallucination capabilities are fragile, easy to damage in training, and do not fully generalize.
Can't help but think that limited "self-awareness" - and I mean that in a very mechanical, no-nonsense "has information about its own capabilities" way - is a major cause of hallucinations. An AI has some awareness of its own capabilities and how certain it is about things - but not nearly enough of it to avoid hallucinations consistently across different domains and settings.
Has anyone had any success with continuous learning type AI products? Seems like there’s a lot of hype around RL to specialise.
There's no known good recipe for continuous learning that's "worth it". No ready-made solution for everyone to copy. People are working on it, no doubt, but it's yet to get to the point of being readily applicable.
Because MCPs solve the exact issue the whole post is about
I asked Perplexity some question for sample UI code for Rust / Slint, it gave me a beautiful web UI, I think it got confused because I wanted to make a UI for an API that has its own web UI, I told it you did NOT give me code for Slint, even though some of its output made references to "ui.slint" and other Rust files, it realized its mistake and gave me exactly what I wanted to see.
tl;dr why dont llms just vet themselves with a new context window to see if they actually answered the question? The "reasoning" models don't always reason.
"Reasoning" models integrate some of that natively. In a way, they're trained to double check themselves - which does improve accuracy at the cost of compute.
> "I will admit, to my slight embarrassment … when we made ChatGPT, I didn't know if it was any good," said Sutskever.
> "When you asked it a factual question, it gave you a wrong answer. I thought it was going to be so unimpressive that people would say, 'Why are you doing this? This is so boring!'" he added.
https://www.businessinsider.com/chatgpt-was-inaccurate-borin...
- Jacob Veldhuyzen van Zanten, respected aviation expert, 1977 teneriffa, brushing off the flight engineers concern about another machine on the runway
Also you could always pick the most likely token in an LLM as well to make it deterministic if you really wanted.
One thing humans tend not to do is confabulate entirely to the degree that LLMs do. When humans do so, it's considered a mental illness. Simply saying the same thing in a different way is not the same as randomly randomly syntactically correct nonsense. Most humans will not, now and then, answer that 2 + 2 = 5, or that the sun rises in the southeast.
You are in my mind rightfully getting pushback for writing "human experts also output tokens with some statistical distribution. "
You have a big opaque box with a slot where you can put text in and you can see text come out. The text that comes out follows some statistical distribution (obviously), and isn't always the same.
Can you decide just from that if there's an LLM or a human sitting inside the box? No. So you can't make conclusions about whether the box as a system is intelligent just because it outputs characters in a stochastic manner according to some distribution.
That shouldn't even be controversial, I don't think?
You wrote "The text that comes out follows some statistical distribution".
At the risk of being over my head here did you mean the text can be described statistically or "follows some statistical distribution". Are these two concepts the same thing? I don't think so.
A program by design follows some statistical distribution. A human is doing whatever electrochemical thing it's doing that can be described statistically after the fact.
Regardless my point was pretty simple, I know this will never happen but I wish tech people would drop this tech language when describing humans and adopt neuroscience language.
Doesn't matter what they think in. A token can be a letter or a word or a sound. The point is that the box takes some sequence of tokens and produces some sequence of tokens.
> You wrote "The text that comes out follows some statistical distribution". > At the risk of being over my head here did you mean the text can be described statistically or "follows some statistical distribution". Are these two concepts the same thing? I don't think so. > A program by design follows some statistical distribution. A human is doing whatever electrochemical thing it's doing that can be described statistically after the fact.
Again, it doesn't matter how the box works internally. You can only observe what goes in and out and observe its distribution.
> Regardless my point was pretty simple, I know this will never happen but I wish tech people would drop this tech language when describing humans and adopt neuroscience language.
My point is neuroscience or not doesn't matter. People make the claim that "the box just produces characters with some stochastic process, therefore it's not intelligent or correct", and I'm saying that implication is not true because there could just as well be a human in the box.
You can't decide whether a system is intelligent just based of the method with which it communicates.
I'd say anybody who writes "the LLM just produces characters with some stochastic process, therefore it's not intelligent or correct" is making an implicit argument about the way the LLM works and the way the human brain works. There might even be an implicit argument about how intelligence works.
They are not making the argument that you can't make up statistical models to describe a box, a human generated text, or an expert human opinion. But that seems to be the claim you are responding to.
I.e. ability to plug in expert data sources
chatGPT (5) is not there especially in replacing my field and skills: graphic, web design and web development. The first 2 there it spits out solid creations per your prompt request yet can not edit it's creations just creates new ones lol. So it's just another tool in my arsenal not a replacement to me.
Such Makes me wonder how it generates the logos and website designs ... is it all just hocus pocus.. the Wizard of OZ?
I don't know about replacing anyone but our UI/UX designers are claiming it's significantly faster than traditional mock ups
Randall Munroe has called this abomination "an insult to life itself". But that might be quoting him out of context.
I don't get why I haven't seen a whole lot of (or any) of these models or tools "self reporting" on "confidence in their answer?"
This feels like it would be REALLY easy; these things predict likelihoods of tokens -- just, you know, give us that number?
On a different note: is it just me or are some parts of this article oddly written? The sentence structure and phrasing read as confusing - which I find ironic, given the context.
No surprise IMO that, generally, online commenters and so-called "tech" companies who tend to be overly fixated on computers as the solution to all problems, are also the most numerous promoters of confidently wrong "AI".
The nature of the medium itself and those so-called "tech" companies that have sought to dominate it through intermediation and "ad services"^1 could have something to do with the acceptance and promotion of confidently wrong "AI". Namely, its ability to reduce critical thinking and the relative ease with which uninformed opinions, misinformation, and other non-factual "confidently wrong" information can be spread by virtually anyone.
1. If "confidently wrong" information is popular, if it "goes viral", then with few exceptions it will be promoted by these companies to drive traffic and increase ad services revenue.
Please note: I could be wrong.
Because “ai” is fallible, right now it is at best a very powerful search engine that can also muck around in (mostly JavaScript) codebases. It also makes mistakes in code, adds cruft, and gives incorrect responses to “research-type” questions. It can usually point you in the right direction, which is cool, but Google was able to do that before its enshittification.
s/AI/LLMs
The part where people call it AI is one of the greatest marketing tricks of the 2020s.
I'm not sure if the comic was AI-assisted or not. AI-generated images do not usually contain identical pixel data when a panel repeats.
Regardless of how the author made the comics, they're very weird.