Pitfalls of premature closure with LLM assisted coding
shayon.dev
shayon.dev
If you put in lazy problem definitions, provide the bare minimum context and review the code cursorily then the output is equally lackluster.
However, if you spend a good amount of time describing the problem, carefully construct a context that includes examples, documentation and relevant files and then review the code with care - you can get some very good code out of them. As I've used them more and more I've noticed that the LLM responds in a thankful way when I provide good context.
> Always ask for alternatives
> Trust but verify
I treat the AI as I would a promising junior engineer. And this article is right, you don't have to accept the first solution from either a junior engineer or the AI. I constantly question the AIs decisions, even when I think they are right. I just checked AI studio and the last message I sent to Gemini was "what is the reasoning behind using metatdata in this case instead of a pricing column?" - the context being a db design discussion where the LLM suggested using an existing JSONB metadata column rather than using a new column. Sometimes I already agree with the approach, I just want to see the AI give an explanation.
And on the trust front, often I let the AI coding agent write the code how it wants to write it rather than force it to write it exactly like I would, just like I would with a junior engineer. Sometimes it gets it right and I learn something. Sometimes it gets it wrong and I have to correct it. I would estimate, 1 out of 10 changes has an obvious error/problem that I have to intervene.
I think of it this way: I control the input and I verify the output.
If it cannot give you a good output with very little prompting, it’s a sign your problem probably isn’t something well known and it probably needs a human touch.
I think this gets to the core of the problem with LLM workflows and why there are so many disagreements about effectiveness
Maybe I overestimate my skills or underestimate how long things would take, but I am constantly feeling like when I try to use AI it takes more time, not less
My suspicion is that if you could create a second version of me, give one copy of them an LLM and one copy solves the problem normally, this would be the case
But many people love these tools and feel more productive, so what gives? The problem it is impossible to really measure because we don't have convenient parallel universe clones to test against. It's all just vibes and made up numbers
There seem to be multiple approaches to working with LLMs. My own personal experience has been that carefully explaining my request, providing specific and highly relevant context (and avoiding irrelevant and distracting context) has lead to significant productivity on my side. That is, it may take me 15 minutes to prepare a really good prompt but the output can save me hours of work. Conversely, if I fire of a bunch of low-effort prompts I get poor results and I end up spending a lot of time in back-and-forth with the LLM and a lot of time fixing up its output.
I thought so too, but sometimes I had better results with one sentence prompt (+README.md) where it delivered the exact thing I wanted. I also had a very detailed prompt with multiple subtasks, all were very detailed +README.md +AGENTS.md and results were very poor.
If you send too much info at once it does seem to confuse the agent, just like if you ask it to do too much all at once. That is yet another property it shares with a junior engineer. It is easy to overwhelm a new contributor to a project with too much information, especially if it isn't strictly relevant.
The rest of the advice in there is sound, but without more specifics I don't know how actionable the section "The spectrum of AI-appropriate tasks" really is.
Basically same problem as copy paste coding, but LLM can (sometimes) know your exact variable names, types so it's easier to forget that you need to understand and check the code.
All via Cursor, some internal tools and Tines Workbench
Last I'd checked, Anthropic would not admit that they were degrading models for obvious scummy business reasons, but they are probably quantizing them, reducing beam search, lowering precision/sampling), etc. because the model goes from being superpowered to completely unusable, constantly dropping code and mangling files, getting caught in loops, doing the weirdest detours, and sometimes completely ignoring my instructions from just one message prior.
t first I wondered if Cursor was mishandling the context, and while they indeed aren't doing the best with context stuffing, the rest of the issues are not context-related.
Engineers think "the LLM can handle the simple code change, but if I ask for too much it'll fall over." Wrong. Modern LLMs can easily handle a 50-line function plus 50 lines of detailed comments explaining assumptions, performance implications, and what changes would invalidate this approach.
But most engineers are either asking for solutions without enough context or failing to ask the LLM to document its assumptions.
Then they're shocked when they have to reverse engineer out that the code assumes 100 users when they have 100k, or why it's doing individual API calls when they needed batch processing.
Most engineers have never seen good comments, so they don't know they can ask LLMs to write them.
The default LLM comment is just English pseudo-code: "this function takes a user ID and sends them a notification." Completely useless. But that's because most engineers have never experienced comments that explain trade-offs, performance implications, or future system evolution.
Writing clear technical explanations is genuinely difficult. Almost no one does it well. So when you ask an LLM for "comments," you get the same terrible pattern you've seen everywhere else.
But you can literally ask for explanations of assumptions, performance characteristics, and scenarios where this approach would break. The LLM handles it perfectly. You just have to know that's even possible. Makes the code review so much easier.
Most engineers don't, because they've never seen it done.
[1] https://peoplesgrocers.com/en/writing/asking-llms-the-right-...
The next improvement may be something like "abstraction isolation" but for now I can vibe code a new feature which will produce something mediocre. Then I ask "is that the cleanest approach?" and it will improve it.
Then I might ask "is this performant?" Or "does this follow the structure used elsewhere?" Or "does this use existing data structures appropriately?" Etc.
Much like the blind men describing an elephant they all might be right, but collectively can still be wrong. Newer, slower models are definitely better at this, but I think rather than throwing infinite context at problems if they were designed with a more top down architectural view and a checklist of competing concerns we might get a lot further in less time.
This seems to be how a lot of people are using them effectively right now - create an architecturr, implement piecemeal.
It can’t even write algorithms that rely on the fact that something is sorted. It needs intermediate glue that is not necessary, etc. massive noise.
Tried single allocation algorithms, can’t do that. Tried guiding to exploit invariants, can’t find single pass workflows.
The public training data is just bad and it can’t really understand what actually good is.
Architecture documentation are helpful too, as you mentioned. They are basically a set of rules and intentions. It's kind of a compressed version of your codebase.
Of course, this means the programmer still has to do all the real work.
If the LLM needs a detailed spec to solve the same problem then you're doing unnecessary work to produce the spec for the LLM first
I imagine it is very slow if you always have to think in a human language and then translate each step into programming language
When people describe being in flow state, I think what is happening is they are more or less thinking directly in the programming language they are writing. No translation step, just writing code
LLM workflows completely remove the ability to achieve that imo
LLM assisted programming, however? – instant flow state. Instead of thinking in code I can think in product, and I can go straight from a pencil sketch to describing it as a set of constraints, and then say, "make sure it's ARIA compliant and responsive", and 95% of the work is done.
I feel similarly about configuration heavy files like Nginx or something. I really don't care to spend my time reading documentation, I'd rather copy paste the entire docs into the context window and then describe what I want in English.
Also good for SQL. And library code for a one off tool or API. And Bash scripting.
I think you are talking about something very different than I am when you say flow state
Aren't the layers the point of abstraction?
In the best case you would have to feed every important information into the context. These are my indexes, this is my function, these are my models. After that the model can find problematic code. So the main problem is go give your model all of the important information, that has to be fixed if the user isn’t doing that part (Obviously that doesn’t mean that the LLM will find a correct problem, but it can improve the results).
Bloated software and unstable code bases abound. This is especially prevalent in legacy code whose maintenance is handed down from one developer to the next, where their understanding of the code base differs from their predecessor’s. Combine that with pressures to ship now vs. getting it right, and you have the perfect recipe for an insipid form of technical debt.
Premature closure is definitely a risk with LLMs but I think code is much less at risk because you can and SHOULD test it. But its a bigger problem for things you cant validate.
I might starting calling this "the original sin" with LLMs... not validating the output. There are many problems people have identified with using LLMs and perhaps all of them come back to not validating.
I would rephrase it - the original sin of llms is not "understanding" what they output. By "understanding" it is meant the "the why of the output" - starting from the original problem and reasoning through to the output solution - ie, the causation process behind the output. What llms do is to pattern match a most "plausible output" to the input. But the output is not born out of a process of causation - it is born out of pattern matching.
Humans can find meaning in the output of LLMs, but machines choke at it - which is why, LLM code looks fine at a first glance until someone tries to run it. Another way to put it is, LLMs sound persuasive to humans - but at the core are rote students who dont understand what they are saying.
But if it’s meant to be a table of contents, it really should be styled like a list, rather than a block quote.