Hacker News Reader

Comment by nomel | Hacker News Reader

> and default to doing what is asked of them instead of what needs to be done.

I don't think it's that simple.

From what I've found, there are "attractors" in the statistics. If a part of your problem is too similar to a very common problem, that the LLM saw a million times, the output will be attracted to those overwhelming statistical next-words, which is understandable. That is the problem I run into most often.

View on HN

It's a constant struggle for me too, both "in the large" and small situations. Using a library which provides special-cased versions of common concepts, like "futures"? You'll get non-stop mistakes and misuses, even if you've got correct ones right next to it, or feed it reams of careful documentation. Got a variable with a name that sounds like it might be a dictionary (e.g. `storesByCity`), but it's actually a list? It'll try to iterate over it like a dictionary, point out "bugs" related to unsorted iteration, and will return `var.Values()` instead of `var` when your func returns a list. Practically every single time, even after multiple rounds of "that's a list"-like feedback or giving it the compilation errors. Got a Clean-Code-like structure in some things but not others? Watch as it assumes everything follows it all the time despite massive evidence to the contrary.

They're rather impressive when building common things in common ways, and a LOT of programming does fit that. But once you step outside that they feel like a pretty strong net negative - some occasional positive surprises, but lots of easy-to-miss mistakes.

I ran into this with cursor a lot. It would keep redoing changes that I explicitly told it I didn’t want. I was coding a game and it would assume things like the players gold should increment at a rate of 5 per tick, then keep putting it back when I said remove it!

Oh sure, the flip side of doing what was asked is doing what is known - choosing a solution based on familiarity rather than applicability. Also a common trait in juniors in my experience

Related, the similarities I see when a human runs out of context window are really interesting.

I do a lot of interviews, and the poor performers usually end up running out of working memory and start behaving very similar to an LLM. Corrections/input from me will go into one ear and fall out the other, they'll start hallucinating aspects of the problem statement in an attractor sort of way, they'll get stuck in loops, etc. I write down when this happens in my notes, and it's very consistently 15 minutes. For all of them, it seems to be the lack of familiarity doesn't allow them to compress/compartmentalize the problem into something that fits in their head. I suspect it's similar for the LLM.

Reply on news.ycombinator.com