0. Use it for research and prototyping, aka throwaway stuff.
2. Use it for studying an existing, complex project. More or less read only or very limited writes.
3. Use it for simple stuff they don't care much about and can validate quickly and reasonably accurately, the standard examples are CLI scripts and GUI layouts.
4. Segment the area in which the LLM works very precisely. Small functions, small modules, ideally they add tests from another source.
5. Boilerplate.
There can be a lot of value in those areas.
When I see stuff like "Amazon saved 4500 dev years of effort by using AI", I know it's on stuff that we would use automation for anyways so it's not really THAT big of a difference over what we've done in the past. But it sounds better if we just pretend like we can compare AI solutions to literally having thousands of developers write Java SDK upgrades manually.
Not just subtle bugs, but unused variables (with names that seem to indicate some important use), comments that don't accurately describe the line of code that it precedes and other things that feel very 'uncanny.'
The problem is, the code often looks really good at first glance. Generally LLMs produce well structured code with good naming conventions etc.
essentially, a longer horizon increases chances of mistakes, increasing time needed to find and fix them. so at one point that becomes greater than the time saved in not having to do it myself
this is why im not bullish on AI agents. task horizon is too long and dynamical
If the task horizon for the LLM is shorter than writing it yourself, this likely means that the task is well defined and has an easy to access answer
For this type of common, well defined task we shouldn't be comparing "how long it takes for the LLM" against "how long it takes to write"
We should be comparing against "how long it takes to find the right answer on SO"
If you use this metric, I bet you the best SO answer, which is also likely the first google result, is just as fast as the LLM. Maybe faster
they have emergent capabilities, like "translating" instructions/questions in X to the probable answers in Y, but i think people are getting way way ahead of themselves with those. these things still fundamentally cant think, and we can try to mimic thinking with scaffolding but then your just going to learn the bitter lesson again