Hacker News Reader

Comment by csmantle | Hacker News Reader

> Takes a few minutes to get the answer [...]

... then waste a few hundred minutes being misled by hallucination. It's quite the opposite of what "cracking open the code" is.

View on HN

I ran into one of the most frightening instances of this recently with Gemini 2.5 Pro.

It insisted that Go 1.25 had made a breaking change to the filepath.Join API. It hallucinated documentation to that effect on both the standard page and release notes. It refused to use web search to correct itself. When I finally (by convincing it that is was another AI checking the previous AIs work) got it to read the page, it claimed that the Go team had modified their release notes after the fact to remove information about the breaking change.

I find myself increasingly convinced that regardless of the “intelligence” of LLMs, they should be kept far away from access to critical systems.

I've found that when any of these agents start going down a really wrong path, you just have to start a new session. I don't think I've ever had success at "redirecting" it once it starts doing weird shit and I assume this is a limitation of next-token prediction since the wrong path is still in the context window. When this happens I often have success telling it to summarize the TODOs/next steps, edit them if I have to remove weird or incorrect goals, and then paste them into a new session.

Like social media, they'll seem benign until they're inervated the populace and start a digital fascism.

Not to forget the energy and computational power wasted to get that answer as well. It’s mindboggling how willingly some people will let their brain get degenerate by handing out shallow debugging tasks to LLMs.

You could look at it as wasting your brain on tasks like that. You start off with a full cup of water each task takes a portion. Farming out thought to an llm can allow you to focus on the next task or the overall before your cup is empty and you need to rest.

The average query is equivalent to 1.1m of driving an electric car. It’s fine, really.

> including gathering the code

LLMs are very reliable when asked about things in their own context window, which is what I recommended.

I'm increasingly convinced that most of the people still complaining about hallucinations with regard to programming just haven't actually used any of the tools in more than a year or two. Or they ran into a bias-confirming speedbump and gave up. Agents obviously hallucinate, because their default and only mode is hallucination, but seeing people insist that they do it too much to be useful just feels like I'm reading an archive of HN from 2022.

Personally I think they are useful, but in a much narrow way than they are often sold as. For things I'm very familiar with, they seem to reduce my productivity by a good chunk. For things I don't want to do like writing some kinds of tests it's probably about the same, but then I don't have to do it, which is a win. For things I'm not very familiar with it probably is at least 2x faster to do with LLM, but that tends to diminish quickly. For example, I recently vibe coded a website using NextJS without knowing almost anything about it. Incredibly fast to get started by applying my existing knowledge of other systems/concepts and using the LLM to extend it to a new space. A week or so of full time work on it later I'm at the point where I know I can get most things done faster by hand, with the occasional LLM detour for things I haven't touched before

It depends on the model knowledge base and what you are trying to do. Something modern with the Buffalo framework in golang many hallucinations. A php blog written in 2005 no hallucinations.

Reply on news.ycombinator.com