Making o1, o3, and Sonnet 3.7 hallucinate for everyone

A good prompt. You don’t just ask it. You tell it how to behave and give it a shot load of context

I use LLMs for writing generic, repetitive code, like scaffolding. It's OK with boring, generic stuff. Sure it makes mistakes occasionally but usually it's a no-brainer to fix them.

> I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up

They can't, they usually just don't understand the code enough to notice the issues immediately.

The perceived quality of LLM answers is inversely proportional to the user's understanding of the topic they're asking about.

GPT can't even tell what its done or give what it knows it should. It's endless, "Apologies, here is what you actually asked for ..." and again it isn't.

A language like Golang tries really hard to only have _one_ way to do something, one right way, one way. Just one way. See how it was before generics. You just have a for loop. Can't really mess up a for loop.

I predict that the variance in success in using LLM for coding (even agentic coding with multi-step rather than a simple line autosuggest or block autosuggest that many are familar with via CoPilot) has much more to do with:

1) is the language a super simple, hard to foot-gun yourself language, with one way to do things that is consistent

AND

2) do juniors and students tend to use the lang, and how much of the online content vis a vis StackOverflow as an example, is written by students or juniors or bootcamp folks writing incorrect code and posting it online.

What % of the online Golang is GH repo like Docker or K8s vs a student posting their buggy Gomoku implementation in StackOverflow?

The future of programming language design has AI-comprehensibility/AI-hallucination-avoidance as one of the key pillars. #1 above is a key aspect.

> LLMs. I really don't get how so many people can get LLMs to code most of their tasks without those issues permanently popping up.

You write tests in the same way as you would when checking your own work or delegating to anyone else?

I've used it for some smaller greenfield code with success. Like, write an Arduino program that performs a number of super-sampled analog readings, and performs a linear regression fit, printing the result to the serial port.

That sort of stuff can be very helpful to newbies in the DIY electronics world for example.

But for anything involving my $dayjob it's been fairly useless beyond writing unit test outlines.

They are good at (combining well-known, codeforces-style) algorithms; often times I don’t care about the syntax, but I need the algorithm. LLMs can write pseudocode for all I care but they tend to get syntax correct quite often

I have made large projects using Claude, with success. I know what I want to do and how to do it, maybe my prompts were right.

What TFA was talking about didn't really seem like an hallucination - just a case of garbage in-garbage out. Normally there are more examples of good/correct data in the training set than bad, so statistically the good wins, but if it's prompted for something obscure maybe bad is all that it has got.

Common coding tasks are going to be better represented in the training set and give better results.

Probably by coding things that are bery mainstream and have already been fed to the LLM a thousand times from ripped off projects.

Here's the secret to coding with an LLM: don't expect it to get things 100% correct. You will need to fix something almost every time you use it to generate code. Maybe a name here, maybe a calculation, maybe a function signature. And maybe you won't spot the issue until later.

You still "used" an LLM to write the code. And it still saved you time (though depending on the circumstances this can be debatable).

That's why all these people say they use LLMs to write lots of code. They aren't saying it did it 100% without any checking and fixing. They're just saying they used it.

The trick to coding with LLMs is not caring if the code is correct.

Sometimes I don't know anything (relatively) about the topic and I want to get a foundation so I don't care whether the syntax or the code is valid as long as it points me in the right direction. Other times I know exactly what I want and I just find that instructing LLM specifically what output I expect from it just helps me get there faster.

Any time I ask anything, LLMs are often bad.

inb4 you just aren't prompting correctly

Yeah this is so common that I've already compiled a mental list of prompts to try against any new release. I haven't seen any improvement in quite a long while now, which confirms my belief that we've more or less hit the scaling wall for what the current approaches can provide. Everything new is just a microoptimization to game one of the benchmarks, but real world use has been identical or even worse for me.

It tried to convince me that it is possible to break out of outer loop in C++ with `break 'label` statement placed in nested loop. No such syntax exists.

Well, it makes sense. The smaller the niche, the lesser weight in the overall training loss. At the end of the day, LLMs are (literally) classifiers that assign probabilities to tokens given some previous tokens.

I once asked Perplexity (using Claude underneath) about some library functionality, which it totally fabricated.

First, I asked it to show me a link to where it got that suggestion, and it scolded me saying that asking for a source is problematic and I must be trying to discredit it.

Then after I responded to that it just said “this is what I thought a solution would look like because I couldn’t find what you were asking for.”

The sad thing is that even though this thing is wrong and wastes my time, it is still somehow preferable to the dogshit Google Search has turned into.

I think a lot of these issues could be avoided if, instead of just a raw model, you have an AI agent which is able to test its own answers against the actual software… it doesn’t matter as much if the model hallucinates if testing weeds out its hallucinations.

Sometimes humans “hallucinate” in a similar way - their memory mixes up different programming languages and they’ll try to use syntax from one in another… but then they’ll quickly discover their mistake when the code doesn’t compile/run

My rule of thumb is: is the answer to your question on the first page of google (a stackoverflow maybe, or some shit like geek4geeks)? If yes GPT can give you an answer, otherwise not.

Semi related: when I'm using a dict of known keys as some sort of simple object, I almost always reach for a dataclass (with slots=True, and kw_only=True) these days. Has the added benefit that you can do stuff like foo = MyDataclass(*some_dict) and get runtime errors when the format has changed

> Any time I ask anything remotely niche, LLMs are often bad

As soon as the AI coder tools (like Aider, Cline, Claude-Coder) come into contact with a _real world_ codebase, it does not end well.

So far I think they managed to fix 2 relatively easy issues on their own, but in other cases they: - Rewrote tests in a way that the broken behaviour passes the test - Fail to solve the core issue in the code, and instead patch-up the broken result (Like `if (result.includes(":") || result.includes("?")) { /* super expensive stupid fixed for a single specific case */ }` - Failed to even update the files properly, wasting a bunch of tokens

This, thank you. It pisses me off to no end when people pretend LLMs are smart. They are nothing but a well trained random text generator.

Seriously, some these conversations feel like interacting someone who believes casting bones and astrology are accurate. Likely because in both cases they are a result of confirmation bias.

Similar the thing at the end with "I find it endearing". I mean, the author feels what he feels, but personally I find this LLM behavior disgusting and depressing.

Same as you and me, really.

People use the general impressiveness of language produced by humans as a proxy measuring their intelligence, that’s how LLMs became AI.

The amount of code out there lets LLMs learn on examples of a lot of tasks and pass SWE benchmarks. Smart autocomplete and solved problem lookup has value. Even if it doesn’t always work correctly, and doesn’t know what it knows.

And for non-programmers they’re indistinguishable from programmers. They produce working code.

It’s easy to see how a product manager or a designer or even a manager that didn’t code for years in a large company can think they’re almost as good as devs.

> All they do is regurgitate text which resembles a continuation of what came before, and sometimes—but with zero guarantees—that text aligns with reality.

You mean like us? Because it takes many runs and debug rounds to make anything that works. Can you write a complex code top-to-bottom in one round, or do you gradually test it to catch bugs you have "hallucinated"?

Both humans and LLMs get into bugs, the question is can we push this to the correct solution or get permanently stuck along the way? And this depends on feedback, sometimes access to a testing environment where we can't let AI run loose.

> because they reveal themselves the second you try to run that code.

In dynamic languages, runtime errors like calling methods with inexistent arguments etc only manifest when the block of code containing them is run and not all blocks of code are run at every invocation of the programs.

As usual, the defenses against this are extensive unit-test coverage and/or static typing.

Where the GAI hallucinates an api it is not always easy to find out if it exists given different versions and libraries for a given task, it can easily waste 10 mins trying to find the promised api, particularly when search results also include generated ai answers.

Also there are plenty of mistakes that will compile and give subtle errors, particularly in dynamic languages and those which allow implicit coercion. Javascript comes to mind. The code can easily be runnable but wrong as well (or worse inconsistent and confusing) and this does happen in practice.

Everything they do is hallucination, some of it ends up being useful and some of it not. The not useful stuff gets called confabulation or hallucination but it's no different from the useful stuff, generated the same exact way. It's all bullshit. Bullshit is actually useful though, when it's not so wrong that it steers people wrong.

Not necessarily. While this may happen sometimes, fundamentally hallucinations don't stem from there being errors in the training data (with the implication that there would be no hallucinations from models trained on error-free data). Hallucinations are inherent to any "given N tokens, append a high-probability token N+1"-style model.

It's more complicated than what happens with Markov chain models but you can use them to build an intuition for what's happening.

Imagine a very simple Markov model trained on these completely factual sentences:

- "The sky is blue and clear"

- "The ocean is blue and deep"

- "Roses are red and fragrant"

When the model is asked to generate text starting with "The roses are...", it might produce: "The roses are blue and deep"

This happens not because any training sentence contained incorrect information, but because the model learned statistical patterns from the text, as opposed to developing a world model based on physical environmental references.

So anyone can make up some random syntax/fact and post it once, and in some cases the model will take it as truth. I don't know if there's a widely agreed-on definition of "hallucination", but if this isn't one, then the distinction is meaningless imo.

That's not true though? Even the original post that has infected LLMs says that the code does not work.

So nothing is a hallucination ever, because anything a LLM ever spits out is somehow somewhere in the training data?

> It’s repeating a pattern in its training data, (…) presented in that training data (…) as being the solution to the problem.

No, it’s presented in the training data as an idea for an interface - the LLM took that and presented it as an existing solution.

Nope there's no attack here.

The training data is the Internet. It has mistakes. There's no available technology to remove all such mistakes.

Whether LLMs hallucinate only because of mistakes in the training data or whether they would hallucinate even if we removed all mistakes is an extremely interesting and important question.

Yet another example how LLMs just regurgitate training data in a slightly mangled form, making most of their use and maybe even training copyright infringement.

Every LLM hallucination comes from some patterns in the training data, combined with lack of awareness that the result isn’t factual. In the present case, the hallucination comes from the unawareness that the pattern was a proposed syntax in the training data and not an actual syntax.

This sort of hallucination happens to me frequently with AWS infrastructure questions. Which is depressing because I can't do anything but agree, "yeah, that API is exactly what any sane person would want, but AWS didn't do that, which is why I'm asking the question".

Also, sometimes a flag does exist, but the example places it incorrectly, causing the command to reject it. Or, a flag used to exist but was removed in the latest versions.

I like your thinking :)

Only if you wanna optimize exclusively for LLM users in this generation.

I agree completely… Usually when I catch it doing this kind of hallucination, it's inventing an API or syntax that is far more clear and intuitive than the actual syntax.

Sometimes that's the case but frequently the thing doesn't exist because of more complex issues. Not every programming language is PHP or JavaScript :-)

Sorry about that. This is a default rails 8 setting, removed the blocker.

It's very easy. I've done this by accident. One of my side projects helps users price the affordability of a particular kind of product. When I ask various LLMs "can I afford X" or "how much do I need to earn to buy X", my project comes up as a source/reference. I currently manually crawl retailers for the MSRP so these numbers are usually months out of date!

Probably because the coworker's question and the forum post are both questions that start with "How do I", so they're a good match. Actual code would be more likely to be preceded by... more code, not a question.

Maybe because the response pattern-matches other languages’s?

What's the difference?

It's more about how you use them. Asking a general-purpose model like ChatGPT precise k8s questions might prove counterproductive; however feeding the entire k8s documentation into an LLM like Gemini and asking questions that way is invaluable. Not just the documentation, but your entire cluster config. Like you said, "blindly trusting LLMs, you'll find yourself in trouble" this is true, but the same can be said for StackOverflow or any other resource. Sifting through StackOverflow to find the exact answer to your question (and then understanding the answer, and hoping that it pertains to your environment, and version etc) is much less efficient when you can ingest the entire docs, your config, your environment, and your question, and have it spit out exactly what you need in whatever format you need. You can even web search with multiple questions derived automatically from your main question, to gather multiple sources which are aggregated and referenced in the answer so you can easily cross-check for hallucinations. StackOverflow isn't even as easy to fact check as LLMs considering the in-line sources.