> LLMs are a lot better than my backend engineers who don't even try but not that much better than someone who's skimmed the docs.

LLMs are basically at the skill-set of "Googled something and tried it", which for a lot of basic things is mostly what everyone does.

If you can loop back the results of the trial & error back to the model, then it does do a pretty good simulation of someone feeling their way through a problem.

The loop is what makes the approach effective, but a model by itself cannot complete the process - it at least needs an agent which can try the recommendation (or a human who will pull the lever).