Hacker News Reader

Comment by jeroenhd | Hacker News Reader

A frightening amount of people have no idea how AI tools work, even those that should know better. I have seen senior software developers fall for the mistake of believing an LLM output when it spews bullshit about how its own memory or restrictions work.

LLMs will listen to you and follow your instructions and restrictions most of the time, which seems to be enough for people to believe that they will every time. I've come to terms with the impact slop coding will have on most software jobs in the future, but seeing seemingly intelligent people fall for lies and fantasies concocted by an LLM is making me more and more uncomfortable with the direction we're all heading in.

View on HN

> LLMs will listen to you and follow your instructions and restrictions most of the time, which seems to be enough for people to believe that they will every time.

It's called automation bias. If something works 90% of the time the human mind will extrapolate that to be 100%. That's just how humans work.

https://en.wikipedia.org/wiki/Automation_bias

LLMs are the first massively popular type of computer programs which actively trying to break half a century worth of human training, which basically distills to "computer programs are highly deterministic and if they work, they are outputting correct predictable results every time" (I'm talking about average population subconscious opinion here, no need to list exceptions). Average person still can't comprehend how LLM output is random all the time and how the identical query to the same program version will produce variable results again and again.

I wonder what will happen after our benevolent prophets St.Sam and St.Dario will succeed in re-training humanity and break this collective expectation of program correctness. I guess they didn't even think about that.

Are we all heading in that direction?

I know it may seem like that reading HN but LLMs are not necessary for writing software, they might be a useful adjunct to it, but they do not have to be central to it (and Id argue they shouldn’t be).

We don’t have to head in this direction of using LLMs for most development at all.

There’s an aspect of extrapolation in the perception spike of the Dunning–Kruger effect.

In the same way smart people, doctors etc, can be better victims for scams I think tech skills can really give the wrong impression of how transformers and LLMs work. If someone has decades of relational database experience all their assumptions will be coloured towards data existing in the model accessible in a rational manner.

Reply on news.ycombinator.com