Hacker News Reader

Comment by symfrog | Hacker News Reader

The closer you get to releasing software, the less useful LLMs become. They tend to go into loops of 'Fixed it!' without having fixed anything.

In my opinion, attempting to hold the hand of the LLM via prompts in English for the 'last mile' to production ready code runs into the fundamental problem of ambiguity of natural languages.

From my experience, those developers that believe LLMs are good enough for production are either building systems that are not critical (e.g. 80% is correct enough), or they do not have the experience to be able to detect how LLM generated code would fail in production beyond the 'happy path'.

View on HN

> The closer you get to releasing software, the less useful LLMs become.

Which is _always_ the case with these things, honestly. Remember Ruby on Rails? Make a Twitter clone in half an hour by just writing some DSL! Of course, in reality Rails was _not_ a productivity revolution, and making _real_ software which had to be operated at scale and maintained, and work properly, in it wasn't much easier than it had been previously.

The amount of "apps" I've had dumped on my team that are everything from un-releasable to deployed on some random shit-cloud we haven't approved (vercel comes up a lot). If you needed hand holding to release things or had to throw software over the fence to others to "productionise" etc then you probably don't know what you're talking about.

This is not my experience with claude code. It does forget big picture things but if you scope your changes well it’s fine.

I would estimate that out of every 200 lines of code that Claude Code produces, I notice at least 1 issue that would cause severe problems in production.

In my opinion these discussions should include MREs (minimal reproducible examples) in the form of prompts to ground the discussion.

For example, take this prompt and put it into Claude Code, can you see the problematic ways it is handling transactions?

---

The invoicing system is being merged into the core system that uses Postgres as its database. The core system has a table for users with columns user_id, username, creation_date . The invoicing data is available in a json file with columns user_id, invoice_id, amount, description.

The data is too big to fit in memory.

Your role is to create a Python program that creates a table for the invoices in Postgres and then inserts the data from the json file. Users will be accessing the system while the invoices are being inserted.

---

What he’s saying is split this up into multiple tasks to create the table, insert the data etc

Isn’t that the hard part? If the tasks are small enough and well defined, where’s the win over just writing the code right there and then?

Well claude can also refine it into smaller tasks and that’s where you can fix those major problems in production issues.

You can use an LLM to generate that list of tasks.

And how does a new grad that's never actually programmed know whether that list of tasks makes sense?

It’s the hard part which is why these tools are so great, the writing of code was the tedious part

And that's why you ask for a high level plan for something like that before you let the agent write any code. Then you review the plan for flaws, revise it, and prompt the system to fill out more details for each step. Repeat as necessary. Yes it's slow, but it's the best way of using this "glorified autocomplete" to ease and speed up real work.

People that have never written their own code won't know what the flaws are.

Those people can ask Claude to review the flaws for them.

Then they won't know if it's accurate or missing something.

Oh good point.

Yes, but knowing how to scope your changes requires a lot of expertise.

Reply on news.ycombinator.com