Hacker News Reader

Comment by Jdstanhope | Hacker News Reader

I have been using Claude to build a clone of Bash in Rust (https://github.com/jdstanhope/huck). It has been goin going on now for 81 days, 2600 commits and I have covered a good chunk of the features such that I can source by .bashrc with a number of extensions. It has been doing pretty good but it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.

That being said, I am still guiding it on a regular basis and only recently let it run open ended trying fixing small to medium size bugs. The next stage will be try let it run in a loop trying to fix the 100 issues it has identified as well as getting most of the bash test suite to pass.

View on HN

Very cool, but I'm skeptical. Over the last year, every Cursor/Claude/Codex project I've worked on devolves into shitty, grafted-on, duct-tape and bandaid'ed architecture, with "fixes" just working around broken code, and tons of duplication that lead to whack-a-mole. The system works at first, but then bugs start piling up and eventually take over. Everyone I talk to shares the same experience. Not to mention that if I, human overseer, don't course correct it many times a day, it goes off into Neverland.

Yet I see comments and articles like above every week, where someone says "Yeah, I just left my agent run for a few nights, and then it made this awesome tool."

There is a huge gap here, but I can't pinpoint it yet. Either I (and everyone I talk to) is Doing It Wrong, or your code is actually flawed at its core and bugs will take exponentially longer to fix and never converge, or... I just need to let my agent run for 3 months??

The only way I've found to make it work on long term AI-code-only projects is to have it constantly refactor, with the axes of refactoring usually provided by me.

Yeah, i basically guide it by writing docs, tests, then code. Then try to get it looping that way. It occasionally happens reliably. If it can reliable prune context itd be pretty close to autonomous. If it achievef that, id sit down and plan out much larger scopes but for now, its just feat->docs->tests|code->loop till done.

> tons of duplication that lead to whack-a-mole

The future is duplication. Code is no longer meant to be read by humans but by machines.

How near/far is that future ?

Cuz what if a human has to intervene as the LLM can’t get it right?

Assuming a human then has till be able to comprehend it?

> Code is no longer meant to be read by humans but by machines.

That isn't true. Everyone knows it and you know it. So why would you make a statement like this? Does it make you feel like a "thought leader"? This isn't X or LinkedIn. Bulshitting here just makes you look like a fool.

If that's true (I very much doubt it), then have the LLMs generate something like WASM binary and run that everywhere instead. No need for human readable languages. Or if LLMs are not that good at binary format, then some textual representation optimized for LLMs instead of humans that compiles to WASM, the JVM or Rust binary.

No need for multiple programming languages anymore. But again, doubt that happens.

Let them. I want to see the faces of middle management when the machine stops doing 'beep', and starts making expensive sounds instead. ;)

It’s not human readability I care about. The problem with duplicate code is that the agent will fix one instance, but leave the others buggy. Hilarity ensues.

That doesn't happen with a proper test suite. Give guardrails and it's not an issue.

Please check out the code and let me know what you think. I have driven a few large refactorings that I don't think Claude would have done on its own. The two big ones were related to handling tokenization and parsing (Claude's original code had circular dependencies and would scan forward in the lever instead of using the parser to determine structure) and the second was for using subshells for command substitution instead of inline and having dual FD and buffer handling.

The current project I am developing with AI is so bloated that I've spent more time and tokens cleaning up the repo (to the tune of tens of thousands of lines removed) than actually creating it in the first place. AI is very very happy to create a mess.

As few people have mentioned you have to guide the AI to clean up stuff, it doesn't seem to choose that on its own. If prompted it can come up with a lot of things to clean up.

Yeah, it always wants to add another layer or append another fix. It almost never goes back and fixes the original thing in place.

My experience exactly. I think we’re doing great, but about 3 weeks in I realized it’s an unworkable mess that keeps breaking, and I spend the next two weeks cleaning it up or rebuilding it

I've been doing a lot of experimenting with this with a c++ compiler project[0]. I definitely experienced a lot of challenges, but I do believe it is possible to end-up in a consistent space by putting in some very careful guardrails. I covered some of the things i learned in the follow on blog posts. My more recent runs have been producing better code after continuing to improve harnesses, tests, and prompts. New blog post coming soon on v2 and v3 reruns. I'll be kicking off some v4 runs soon. Here is a teaser showing the later runs: https://storage.googleapis.com/ralph-run-viewer-zippy-960/in...

[0]: https://medium.com/@vishvananda/i-spent-2-billion-tokens-wri...

IME you can get good results if:

1. You have very good tests. The agent can write them, but you have to prompt it to do that, and some human input may be needed here.

2. You have a good, modular architecture. Again, this is something that needs to be prompted and/or developed up front. Agent work can scale if you divide and conquer, so that any given task has a well-defined scope, limiting the amount of context needed to handle it well. If you let an agent just build a big pile of code, you’ll have issues.

Once you have the above, you’re more likely to be able to just let the agent run.

This is why you hear about things like SQLite being ported to Rust (by multiple groups!) It has a comprehensive unit test suite and a good existing architecture. Such porting projects are benefitting from already having done the upfront work that I mentioned, plus they have the original source code as a reference implementation.

Because of that, most of what I do involves a lot of interaction with the agent. That’s because I’m typically working on complex existing systems, that don’t necessarily have the above properties, and agents aren’t at the point where they can handle that autonomously.

This is exactly the case for "porting" Bash to Rust. There is a huge amount of existing tests that can be leverage and when in doubt you can run the same code in bash and Huck and compare the result and then fix the issue.

If you just say “implement this” you’re going to have a bad time after a certain point.

You really need to add yourself as a human in loop to be in the middle of design choices. That is, ask the model for a plan, what the trade offs are, should parts of the code be refactored before the next feature, etc.

Also, there’s an element of reading the code and ordering refactors. I’ve noticed that for embedded code codex loves to to do everything in a main.c and too many compiler defines. Asking it to propose a refactoring into modules helps a lot.

It’s just like writing real code, if you don’t do design up front and don’t aggressively refactor as the requirements change, it becomes a mess.

> Not to mention that if I, human overseer, don't course correct it many times a day, it goes off into Neverland.

Doesn't that happen to a human engineering team if you don't course correct it many times a month (which in code output is similar to an LLM in a day)?

Can't tell if this comment is serious. No, there are individual people who are capable of writing operating systems, browsers, source control systems, complex video games, and programming languages with no external guidance, design reviews, or anything. They are fully autonomous intelligent beings with good taste, powerful programming ability, and the vision to set long term and short term goals, craft plans, and execute without outside supervision.

There are also small teams of experts who need little to no supervision, and will NOT devolve without constant correction. They will do things like look for an existing function before writing a new one, or use good judgement when trying to find a library, rather than reinventing the wheel.

> Over the last year, every Cursor/Claude/Codex project I've worked on devolves into shitty, grafted-on, duct-tape and bandaid'ed architecture

I've noticed this pattern as well when people decide to vibe code a large project.

They usually get their MVP quickly, but as soon as they want a new feature or to fix a bug, the LLM takes the shortest possible route to a solution. The problem is that after about a dozen rounds of "the honest fix" and "load-bearing code" (insert your favorite claudeism) it's a tangled mess of enterprise-grade spaghetti.

I think it's because the people full-on vibe coding just don't have the experience or time to design large scale systems and anticipate what might cause problems if they want to change something later.

IMO coding agents can be useful, but the only safe place is the manageable changes you would design up and hand off to a junior anyway (but can't because your company decided to outsource years ago).

> it will sometimes spend hours to do few line fix and other times pushback on making big changes despite have over 3000 tests.

you ever worked with human engineers before? :D

I was going to say that it's pretty much exactly like working with an experienced software engineer ... except it never seems to take a step back and look at the bigger picture. It will happily fix the same class of bugs over and over without realizing a bigger change is required.

Wait, are you talking about llms or people?

It's sad that LLMs are learning to devalue human life from people like you.

Step 1: Develop human-like minds

Step 2: Dehumanize them constantly

LLMs don’t have human-like minds in any perceptible sense though.

Do. Not. Anthropomorphize. The Woodchipper.

"Not everything needs to be an anime girl"

Funny how that never made it to the rules of the internet.

The rules of the internet are for things that are true.

Are you referring to Larry Ellison?

I was about to say as well, I don’t think I’d be interested in ever using this project but now I definitely never will. I wonder if they are vibe coding this for humans but have disdain for humans, why make it for humans?

That’s a direct result of how they’re fine tuned. RLHF and other mechanisms reward quick, locally correct answers that solve the user’s immediate problem. A reward for a solution like "take a two-day pause, rip out half the modules, and rewrite the core" just straight-up doesn't exist in training datasets

I find most of the time it would be fine on its own, but every now and again you avoid a really bad degradation through guidance. Some kinds of big change definitely problematic.

Reply on news.ycombinator.com