LLMs will listen to you and follow your instructions and restrictions most of the time, which seems to be enough for people to believe that they will every time. I've come to terms with the impact slop coding will have on most software jobs in the future, but seeing seemingly intelligent people fall for lies and fantasies concocted by an LLM is making me more and more uncomfortable with the direction we're all heading in.
In the same way smart people, doctors etc, can be better victims for scams I think tech skills can really give the wrong impression of how transformers and LLMs work. If someone has decades of relational database experience all their assumptions will be coloured towards data existing in the model accessible in a rational manner.
I know it may seem like that reading HN but LLMs are not necessary for writing software, they might be a useful adjunct to it, but they do not have to be central to it (and Id argue they shouldn’t be).
We don’t have to head in this direction of using LLMs for most development at all.
It's called automation bias. If something works 90% of the time the human mind will extrapolate that to be 100%. That's just how humans work.
I wonder what will happen after our benevolent prophets St.Sam and St.Dario will succeed in re-training humanity and break this collective expectation of program correctness. I guess they didn't even think about that.