Hacker News Reader

Comment by gchamonlive | Hacker News Reader

I think it's not only a matter of token efficiency. If you don't know what you are doing development will eventually crawl to a halt invariably.

It's the compound counter-probability of success, so even a 99% efficient model will in time accumulate so much error that without conscious cleanup and steering, it becomes really unlikely really fast that anything could be changed in the code without affecting something else, no matter how many tokens you throw at it. It's the collapse of a complex system under the weight of sheer uncertainty of what the system actually does.

View on HN

We’re at 99% for a lot of stuff today, you get a third nine from council reviews and labs have another one or two nines in the pipeline. At five nines your task length horizon extends far beyond the current frontier model release cadence. More out of distribution tasks lose a nine or two, still revolutionary. You can get an extra nine from a good set of skills around slicing and distributing work according to model capabilities.

The llm are improving though maybe a year from now they can use it to fix the code

Maybe, it'll be exciting to see, and I'm all about accessibility, but in this case I also don't think it's about model capability or intelligence, it's the low information to noise ratio in the codebase. There just won't be enough information in the code itself to know what to fix. Fix how? What should it do? I'm really not sure you can reconstruct intention from a codebase created unsupervised.

1. Implement feature and write tests for the code

2. Make sure tests pass

3. <Every now and then> Review code for quality and fix - make sure tests pass.

4. Go to #1

Overly simplistic? Yes. But I would wager that this can go a long way, even for vibe coders.

I keep seeing this but I'm not sure it is effective in the long run *if unsupervised*.

"Review quality and fix" doesn't mean a lot without context.

Does it mean to remove unused features and simplify the underlaying code? Does it mean changing the data structures to better support future development? Does it mean improving performance because of bottlenecks?

You are supposed to tell an LLM what your codebase needs, but if you just vibe code without knowing the code, "review quality and fix" will have unexpected results

It's glaring to see who's still buying the positivists AI hype and who's done actual work with it.

AI is amazing, but people need to realise meaning and intention can't exist in a vacuum.

Commit the intention as specs. If tokens/intelligence become that much cheaper over time, then "throwing away the code" to start again becomes feasible.

Yeah, but not quite. I've done this with a project of mine, wrote it in python so it's easy and quick to read and debug, saved every work item and decision, done everything through agents so there was logs to go back to, and when I tried reimplementing in elixir everything fell apart.

Even after documenting decision and specs you need somehow to replay these in the correct sequence after you've validated these specs are still updated. Imagine you spec a feature, it works well, but in an edge case while doing a separate work you see something wrong, will you stop, fix and update the related spec? You will trust the AI will do this, and you guessed right, the counter-probability of success also has effect here, so eventually you will do undocumented changed in the codebase that won't reflect in the spec.

Now imagine all this but in the hands of someone that is an expert in their field but has zero notion what we are talking about here. Just look at the state of packages in R, the programming language, you'll see that technical competency and intelligence don't translate immediately to efficiency in a programming role.

A few days ago somebody linked to their own project: https://ljtn.github.io/epiq/

I haven't had time to dive into it yet, but I think it might structure things in the way you want.

Thanks! Will look into it.

This is something I didn't think about. Have a tech illiterate friendly harness that will keep asking technical questions the mainstream user won't be aware they needed be addressed, until there is enough evidence to either start an implementation or outright reject the project with suggestions where the user might look into to better prepare for another session.

Can confirm, I'm currently using Opus 5.5 to fix some Opus 4.5 slop from around this time last year.

Fixing code is not the same as fixing a fundamentally broken architecture. The latter requires understanding that the architecture is broken in the first place and that understanding comes from experience.

And most of the time the architecture is not just broken, it's just meaningless because of so much accumulated error. If it was just broken then other comments would make sense, just throw a more capable model at it, but LLMs can't really reproduce shakespeare out of white noise just yet.

Reply on news.ycombinator.com