Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase.
Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.
That’s a fairly common error, on my part. I may have a wrong file, selected in the Xcode Navigator, and the LLM accepts that, as the problematic file. In other cases, I’m just plain wrong, in my diagnosis. Maybe the error manifests in that file, but is caused, in another one. I have taken to giving the LLM as many involved files as possible, when asking for help.
I found that, as the LLM progressed through my project, learning more and more about it, it caught these errors, but early on, it ran with my wrong input, and gave bad advice.
It still, occasionally, gives bad advice, but that’s just because it was probably trained on bad code. I have found that it gives good advice, more often than not. It’s just incumbent on me, to treat its input as “advice,” and carefully consider it, before integrating it into my own work.
I have just upgraded my account. In order to do this, I had to create a new one, for billing purposes. This means that it needs to start over, learning the project. I am in the process of exporting from the previous account, but I’m not sure how well this will transfer the accumulated knowledge.
LLMs don’t learn though. Are you talking about some sort of architecture.md file collection being built? This files should be just easy to copy?
That's actually a real advantage. Right now, I am getting it to compile a tutorial for the app. Since it's starting from scratch, I can't rely on it remembering much, so I have to "start from the beginning," and give it a lot of information that the other account already had.
I know Claude will maintain its own memory files in ~/.claude, unbidden.
The actual LLM under the hood is static, though - it was trained once and will not self-modify.
IIRC ChatGPT does use "memory" systems to maintain familiarity with a given chat's subject.
I believe that's an important ingredient for creating AI psychosis - human and AI become slowly more unhinged together in a feedback loop until it finally spirals out of control.
Sure, the model can go wrong, but sometimes it's able to realise that and correct its course. Stronger models are better at doing this.
People do exactly the same thing! Haven't you ever wasted a lot of time chasing down a blind alley?
To say the LLM has immutable limits because it only predicts the next token and can't backtrack is like saying we have immutable limits because we can't travel backwards in time. It's a true statement but not particularly relevant or helpful.
Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling.
If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.
For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good.
In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.
That is I very rarely even use a “chat” interface anymore. Under the covers I’m still using the mainline agents but I’m treating them more like black box Unix tools that take text in and spit text out.
My interactions are evolving to have very small llm cores wrapped with old fashioned deterministic programming.
This makes the boundary/verification step more deterministic and less susceptible to bad predictions. But it also has the side effect that I rarely interact with llm output that has “tone” (for instance sycophantic output). That usually only happens when I’m debugging what went wrong, particularly around model upgrade cycles.
Thus is why I think calling out that it is just predicting tokens is relevant. LLM are trained on human written communication. Humans say they found the root cause often, so the LLMs say it often. Even when not true.
Idk if thats true or not, but LLMs mimicking some average speech written pattern isnt crazy to me.
You can argue back and forth about whether LLMs are actually “conscious”, whatever that means, but it’s clear that they can be tremendously effective and useful. They can use language to get stuff done.
Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.
Take yourself out of that loop so you only find out once it's sure.
I keep thinking about the c compiler implementation that anthropic shared earlier in the year that had all the requirements you mention and arguably wasn’t that great.
OBVIOUSLY, the compiler experiment was just a cringe pr stunt. But it has a point: everything works better with a good testing loop, and compilers always have one by thr nature of the work they do
No amount of tokens can come close to my hourly rate.
Hm. Two orthogonal properties! This sounds like a 2x2 matrix!
Let's swap hard/easy around & explore the 4 possibilities...
There are domains with sharp delineation and hard verification; they are not at risk until AI gets much better. Humans operate in these domains by applying tremendous deep thought and subjective judgment - our superpower.
Domains with soft delineation and easy verification are most at risk: "it's a picture of a cat" remains true through a wide range of perturbations - eg. skewing the image or moving it across a pixel or correcting its white balance or even changing the cat. AI music? Lots of domains already solved by AI here but they're also not that meaty.
My prediction is the next interesting stuff will happen where verification is hard but there's no sharp delineation. It's the world of "I'll know it when I see it". Good customer service?
LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.
Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods.
That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.
Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare.
To write an honest answer, it says, "Here's the honest answer", and proceeds from there.
If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker".
This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext.
You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.
In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.