Hacker News Reader

Comment by Someone | Hacker News Reader

> That is not true, and the proof is that LLMs _can_ reliably generate (relatively small amounts of) working code from relatively terse descriptions.

LLMs can generate (relatively small amounts of) working code from relatively terse descriptions, but I don’t think they can do so _reliably_.

They’re more reliable the shorter the code fragment and the more common the code, but they do break down for complex descriptions. For example, try tweaking the description of a widely-known algorithm just a little bit and see how good the generated code follows the spec.

> Sometimes the interpolated detail is wrong (and indeterministic), so, if reliable result is to be achieved

Seems you agree they _cannot_ reliably generate (relatively small amounts of) working code from relatively terse descriptions

View on HN

Neither can humans, but the industry has decades of experience with how to instruct and guide human developer teams using specs.

Usually, you don't want your developers to be coding monkeys, for good results. You need the human developer in the loop to even define the spec, maybe contributing ideas, but at the very least asking questions about "what happens when..." and "have you thought about...".

In fact, this is a huge chunk of the value a developer brings to the table.

And this is usually one of the defining traits of a senior engineer. They understand the tech and its limitations, and thus are able to look around corners, ask good questions, and, overall, provide quality product input.

In other words, prudential judgement.

Programs are a socially constructed artifact that help communicate and express a model (which is perpetually locked in people's heads with variance across engineers; divergence is addressed as the program develops). Determining what should or should not be done is a matter of not just domain knowledge, but practical reason, which is to say prudence, which is a virtue that can only be acquired by experience. It is an ability to apply universal principles to particular situations.

This is why young devs, even when clever in some local sense, are worse at understanding the right moves to make in context. Code does not stand alone. It exists entirely in the service of something and is bound by constraints that are external to it.

This is very much my experience from working with outsourced development. Almost by design, they tend to lack domain expertise or an intimate understanding of the cultures and engineering values of the company they're contracted out to.

This means that they will very quickly help you discover all the little details that seemed so obvious to you that you didn't even think to mention them, but were nonetheless critical to a successful implementation. The corollary to that is, the potential ROI of outsourcing is inversely proportional to how many of these little details your project has, and how important they are.

So far I've found LLM coding to be kind of the same. For projects where those details are relatively unimportant, they can save me a bunch of effort. But I would not want to let an LLM build and maintain something like an API or database schema. Doing a good job of those requires too much knowledge of expected usage patterns working through design tradeoffs. And they tend to be incredibly expensive to change after deployment so it pays to take your time and get your hands dirty.

I also kind of hate them for writing tests, for similar reasons. I know many people love them for it because writing tests isn't super happy fun times, but for my part I'm tired of dealing with LLM-generated test suites being so brittle that they actively hinder future development.

Humans have the ability to retrospect, push back on a faulty spec, push back on an unclarified spec, do experiments, make judgement calls and build tools and processes to account for their own foibles.

which bits of this do you think llm based agents can't do?

The main thing they cannot do is be held accountable for any decisions, which makes them not trustworthy.

This is not correct. They can say "sorry" which makes them as accountable as ordinary developer.

I've found recent versions of Claude and codex to be reluctant in this regard. They will recognise the problem they created a few minutes ago but often behave as if someone else did it. In many ways that's true though, I suppose.

Does it do this for really cut and dry problems? I’ve noticed that ChatGPT will put a lot of effort into (retroactively) “discovering” a basically-valid alternative interpretation of something it said previously, if you object on good grounds. Like it’s trying to evade admitting that it made a mistake, but also find some say to satisfy your objection. Fair enough, if slightly annoying.

But I have also caught it on straightforward matters of fact and it’ll apologize. Sometimes in an over the top fashion…

That's not what accountability is

Accountability: "Something that SWE's run screaming from".

Example: "We should have professional accountability in software"

SWE: "This would bring about the end of the world!!!1!"

The economics of software development have lowered the bar for software engineers: there simply aren't enough people who are good at it (or even want to be), and the salaries are very high, so plenty of people who shouldn't be SWE's are.

I am a software engineer, and I would absolutely love to see more professional accountability in this field. Unfortunately, it would make the cost of software go up significantly (because many many people writing software would be ejected from the industry)

LLM based solutions don’t need to stay dry and warm at night, with a full belly, possibly with their sexual partner with whom they have a drive to procreate.

Ordinary developers get fired for poor performance *all the time*.

Not get stuck on an incorrect train of thought, not ignore core instructions in favour of training data like breaking naming conventions across sessions or long contexts, not confidently state "I completely understand the problem and this will definitely work this time" for the 5th time without actually checking. I could go on.

LLMs by their nature are not goal orientated (this is fundamental difference of reinforcement learning vs neural networks for example). So a human will have the, let's say, the ultimate goal of creating value with a web application they create ("save me time!"). The LLM has no concept of that. It's trying to complete a spec as best it can with no knowledge of the goal. Even if you tell it the goal it has no concept of the process to achieve or confirm the goal was attained - you have to tell it that.

any of them.

Humans also have the ability to introspect. Ultimately, (nearly) every software project is intended to provide a service to humans, and most humans are similar in most ways: "what would I want it to do?" is a surprisingly-reliable heuristic for dealing with ambiguity, especially if you know where you should and shouldn't expect it to be valid.

The best LLMs can manage is "what's statistically-plausible behaviour for descriptions of humans in the corpus", which is not the same thing at all. Sometimes, I imagine, that might be more useful; but for programming (where, assuming you're not reinventing wheels or scrimping on your research, you're often encountering situations that nobody has encountered before), an alien mind's extrapolation of statistically-plausible human behaviour observations is not useful. (I'm using "alien mind" metaphorically, since LLMs do not appear particularly mind-like to me.)

Most companies I've worked for have had 'know the customer' events so that developers learn what the customers really do and in turn even if we are not in their domain we have a good idea what they care about.

Specs are insufficient to guide human developer teams, so I don’t understand the comparison.

When LLMs generate an appropriate program from ambiguous requirements, they do this because the requirements happen to match something similar that has been done previously elsewhere.

There is a huge amount of programming work that consists in reinventing the wheel, i.e. in redoing something very similar to programs that have been written thousands of times before.

For this kind of work LLMs can greatly improve productivity, even if they are not much better than if you would be allowed to search, copy and paste from the programs on which the LLM has been trained. The advantage of an LLM is the automation of the search/copy/paste actions, and even more than this, the removal of the copyrights from the original programs. The copyright laws are what has resulted in huge amounts of superfluous programming work, which is necessary even when there are open-source solutions, but the employer of the programmer wants to "own the IP".

On the other hand, for really novel applications, or for old applications where you want to obtain better performance than anyone has gotten before, providing an ambiguous prompt to an LLM will get you nowhere.

> and even more than this, the removal of the copyrights from the original programs

This seems really strange to me. Can you explain how this is different than just stealing code from other sources, or copying it wholly from open source repos?

You can guide humans, but ultimately the reason senior software developers have been payed large sums of money is that even with specs mostly we have found it works better to have someone with good judgement actually doing the work, otherwise we would have just been using specifications. The question remains open if llm’s can show good judgement, often my experience with claude is that it doesn’t if the problem domain is non-trivial but it’s possible that won’t always be true.

> try tweaking the description of a widely-known algorithm just a little bit and see how good the generated code follows the spec.

this works well for me

anything can be reliable if you have good tests

Reply on news.ycombinator.com