For me, AI tools shine when I know enough about a topic to quickly error check, but not enough where I can code fluently without documentation. I'm sure it will get more useful over time, but that's where it's been for me for the last year or so.
It seems that way if you judge them the same way you would human coders, but they’re different. They might be able to do things that veteran coders can’t without spending days on it, and fail at things that beginners can do in half an hour.
A car might not be able to traverse difficult terrain as well as a horse, but it doesn’t mean the car is not a good mode of transportation.
Also, their eagerness to produce. A good employee (senior or not!), knows when they're at the end of their wits and say so. In areas where LLMs are knowledgeable, but not knowledgeable enough, they'll still produce. They're stuck at unconscious incompetence, kind of like a systematic Dunning-Kruger effect. To solve that, we can either increase competence (moving out the area where this effect is noticeable), or maybe introduce some kind of feedback, self reflection (one could say, some kind of conciousness [0]).
[0] inspired by "Goedel, Escher, Bach"
That doesn’t mean they are unpredictably bad, it means assuming that they behave like humans is a bad way of predicting their behaviour. “Unlike humans” and “unpredictable” are not the same thing. If you spend time working with them, you get a better sense of what they are good and bad at and get better at predicting their behaviour.
The sibling comment says it better - it's about predicability of mistakes, and the effort and knowledge necessary to correct them.
Maybe the next generation of models will solve these problems, who knows.
So if you drive well and can control this "car", you are able to fix that and benefit from the fast "car" a lot.
> or a car that randomly refuses to go on certain roads
This doesn't seem like a good comparison. AI will do anything, it may just be wrong.
I'm fine with ditching the car analogy though.
Anecdotally, I think this behavior is undesirable for most commercial LLM use cases. I have several friends that have complained about Gemini’s “back talking” and prefer ChatGPT’s relative sycophancy.