Comment by mccoyb | Hacker News Reader

Comment by mccoyb | Hacker News Reader

Here's this boiled down:

> A stochastic search process with an executable optimization objective over space of programs S can only maintain or improve the objective

This is superoptimization. We've known this since the 80s (Massalin, STOKE is more recent: https://github.com/StanfordPL/stoke) The only novelty is that the proposer is now way better with LMs.

Further, there's a large number of reasons for software written by agents to be slow:

- LMs still don't do data or hardware-oriented design well out of the box, and therefore if you're engaging in any sort of serious novel work, beyond porting an extremely well-understood program with extremely well-understood workloads, you're going to be spending hours tracking down bad allocation decisions (c.f. why TigerBeetle doesn't use agents), which are often the root of evil (before you'd reach for anything further)

- The knobs you'd need to get serious performance are nearly unreachable in languages which LMs are good at (even Rust requires a discipline that the default language doesn't enforce). When you drop into the lower realms, you're trading consumption context for access to these levers. The levers are also "soft": you find yourself writing a bunch of skills, and tools to try and enforce the discipline.

The reality is to get performant code (quickly) out of an agent, you need to know how to write performant code (and you need to know how to surface the information that you'd use to create a verifier for such a thing to the agent), which 99% of developers do not know in 2026.

Sure, agents can teach you how to do this -- but it's one of these things where iykyk.

Experience: I've poured 10s of billions of tokens into Zig with the best agents and I have the time and space to try these things.

If you want to start learning the discipline, I'd recommend matklad's + TigerBeetle blog -- as well as hardware-oriented design.

> - The knobs you'd need to get serious performance are nearly unreachable in languages which LMs are good at (even Rust requires a discipline that the default language doesn't enforce). When you drop into the lower realms, you're trading consumption context for access to these levers. The levers are also "soft": you find yourself writing a bunch of skills, and tools to try and enforce the discipline.

Hasn’t been my experience at all. The latest LLMs can knock out assembly optimized subroutines and benchmark 100 different variations faster than I ever could dream of.

That's fair for a well-scoped subroutine: what I meant is that if you ask an agent to write a compiler and let it rip for a few days, you are going to be spending a few more days correcting the default behaviors in the distribution, which often do not tend towards hardware-oriented design.

To correct those behaviors, you're going to write tools and skills, and that's going to help, but it is still clear that you are fighting the distribution (today).

Yes, say "Build a compiler" will require you to clean up stuff if you leave the agent for days, but not because of the LLM or the quality of the tool, but because you hardly specified anything, so of course it's gonna make assumptions you need to correct.

If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things, just code fulfilling what the specification says.

Granted, this does require you to not use obviously dumb models, like anything you can run locally today, and at least within reasonable range of SOTA models. But they been able to do this for 6 months or more at this point.

You're making assumptions; OP made no mention of how detailed their spec was.

Indeed, you're right, I did do assumptions, based on what I could understand from it, and also based on my own experience using these tools too. May be their specification is very detailed and exactly as they wish it to be, yet the agent creates something completely else, then probably something else than the specification is wrong, and my comment is pointless :)

Bro, what the fuck do you think I’m doing? Do you think I don’t know about spec driven development?

What is the most complicated thing you’ve built with LM agents? Have you done it with a single spec? How novel was it?

This comment is so laughably “you’re holding it wrong” I can’t respond to you seriously.

> If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things

The set of software that has followed this process is measure zero.

> Bro, what the fuck do you think I’m doing? Do you think I don’t know about spec driven development?

Bro, I don't know you, and have no idea what you do, except for the parts you share in your comment, like talking to a LLM :) If you say "if you ask an agent to write a compiler" and don't expand on that, I'll take your word for it.

I'm glad you got a laugh, and didn't take it so serious :) Not entirely sure why you got so defensive, but it's all good, again I don't know you, sorry if it felt like I called you stupid or something. At least some other people found it helpful I suppose, so in the end it sucks you're unable to get value from a process I've found working for me, hope you'll find something that works out for you in the end, regardless of what it is!

I'm still interested in your claim:

> If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things, just code fulfilling what the specification says.

Are you claiming that you reliably get good software out of agents using this process?

My experiential claim is that I've followed this process for complex pieces of software (and, often, taking significantly longer than a day -- more like several weeks, as if I was writing an academic paper) and they still get it wrong ... even with an "academic paper level spec"

Models: only the best, Fable 5 max, Sol 5.6 max, yada yada -- I'm not convinced this process works for "serious endeavors" (for instance: let's build a novel accelerator compiler or let's build a new incremental linker) where "serious" here means "something where I'll likely be spending multiple weeks working on it"

> Are you claiming that you reliably get good software out of agents using this process?

Yeah, for sure, I tend to be really strict about the overall design, keeping things minimal and only the abstractions needed, nothing more/less.

I don't know what "most complicated software" even is, I've only been writing software for 20 years or so, but mostly around startups, smaller companies and as a consultant/freelancer for companies of all sizes, so I won't even claim to know what "really complicated software" is. What I've done so far ranges from implementations of various ML models, native cross-platform applications, various types of games, simulation software and a bunch of tooling to help me with music production and VFX/animation/3D work.

Some of those I guess I wouldn't even have approached without LLMs, and probably would have taken me years if I did attempt, but following that process I basically end up with one spec, maybe 2K-3K lines in total, including step-by-step how things have to be implemented, then fire off Codex + GPT-5.3-Sol max with "/goal" saying exactly how things should be implemented, verified and so on, and update the specification file as it goes along, and not stop until everything from the specification been confirmed to be implemented correctly. Longest been 4-5 days until the goal is completed.

Always a lot of iterations afterwards of course, as I realize some things I put in the specification ultimately didn't made sense once it's implemented, so revise specification, figure out how to change, fire off "/goal", and repeat this until I have code that is better or at least equal to how I'd write it myself. It was a long time ago this process lead to something completely different being implemented, I usually get exactly what the specification says.

Okay, so we’re using the same process — but your original message seemed to imply a sort of one shot no refinement iterations — which is what I was responding to as unrealistic (e.g. make a spec let goal run artifact is perfect)

Of course, all I’m saying is that you need to refine your sample! For instance: the allocation architecture is not correct, and one has to run a bunch of performance investigations and resolve it.

My responses are intending to convey that I don’t believe this is possible, no matter how good LMs get — and it seems like we are in agreement.

We're mixing two different things here though. You're saying that you would need to correct the compiler because the agent did wrong, I'm saying that you'll need to correct the compiler because your specification will be wrong. The agent does the correct thing, but the correct thing was wrong in some way, if that makes sense?

If your goal with building this compiler was performance, and this wasn't part of the initial specification, and the agent didn't assume it had to, is this what you're saying is a failure on the agents side?

There is no distribution to fight, is my hypothesis at least, if you're just a lot more clear exactly what you expect up front. Hence the whole "To correct those behaviors, you're going to write tools and skills" thing isn't even needed in the first place.

Sorry for defensiveness: no, I know what I'm doing, and I'm careful to move with understanding.

I don't believe the problem is "ah, you didn't write the spec clearly enough" -- which is why I'm asking about your own experiences.

Agreed, IME Fable can churn out decent SIMD kernels optimized for whatever tradeoffs you give it.

> iykyk

A story.

I have a friend, who is - like me - interested in the CRDT / collaborative editing space. He asked ChatGPT to write him a CRDT. Then he grabbed every good CRDT implementation, and asked chatgpt to benchmark and optimise his CRDT, using tricks and techniques from existing hand-optimised CRDTs. He got massive performance gains by doing this - which is really interesting! I think it helped that he had a clear objective function, and chatgpt could look at other projects for ideas on how to optimise.

He proudly boasted that his resulting code outperformed my diamond-types library. I asked him if he was comparing against the native implementation, or the -Oz webassembly build, running in a wasm vm. It was the latter. When he tested it properly, his CRDT was - and is - significantly slower than diamond types. As far as I know, chatgpt still hasn't been able to catch up. I tried myself using fable. Even with reference to my source code, Fable still doesn't understand what I did in diamond types and why. (... Maybe I should document what I did!)

I think his technique itself is solid though. I tried it myself. I asked fable to write a custom binary serialization format & parser. Then optimise. Then optimise, with explicit reference to existing libraries. Optimising with reference to other code made a huge additional difference. It is now nearly as fast as those libraries. (But still not faster than them.)

My takeaway is this: I think LLMs are exceptionally good at reading and understanding code. If you guide them to do so, they're good at profiling and benchmarking. But it seems like they're not very good at coming up with novel optimisations. If you have an obviously slow program (for example, some slop claude wrote), you can often get big speedups by asking it to benchmark and optimise. But if you have a complex, already well optimised codebase, like the zig compiler, claude doesn't seem very good at figuring out novel ways to improve things on its own.

This is good news for the 95% of slow software out there. But bad news for the 5% of us who write fast code already, but want our code to go even faster.

The refutation of your takeaway is autoresearch and similar. They can brute force novel optimizations (and generally achieve superhuman performance) when provided with an appropriate environment.

Of course that doesn't mean they have a human level mental model and associated novel ideas. Brute force can be effective but remains entirely unsatisfying from an academic perspective.

True! How do you set up auto research loops? Are there any special tricks to it?

I don't know if it's the first and it certainly isn't state of the art at this point but I think karpathy/autoresearch is a quintessential starting point.

TBF as an earlier commenter noted it's "just" superoptimization using an LLM as the proposer so there's a lot of relevant prior art.

yep, i think the real LLM superpower is knowing that something has been done before and having access to the code that did it. so much of even novel software includes bits and pieces that have well-optimised existing solutions, and the bot knows those solutions a lot better than i do, and can even pattern match them from the general shape of the problem.

Yes. It’s also excellent at reading large codebases and putting together a picture of what’s going on. I’ve been using it a lot lately to brief me on projects and design decisions. “Look at these two projects. They both solve task X. Write a report about their similarities and differences, and the tradeoffs as a result.” And then I ask followup questions. Saves a ton of time.

Hi Joseph,

A story for a story. I had my CRDT implementation in libdog, which does per-token CRDT weave/diff/merge over a DAG of git blobs. It was written by Claude 4.8 I believe, in several iterations. It was, as you may guess, a piece of neuro-slop that passed the tests by some miracle. Once I had some time to look into it, I used a trick: I supplied it with my article on Chronofolds and some helpful kicks in the butt. It implemented everything correctly on its k-th attempt, k<5. Then I used it with full intensity for three months without thinking twice. Now I have started mass-using it to resolve permalinks in the code. Like, tens of files to chronofold-ize per one commit. It is now showing up in the profile, so I may look into it once more.

Conclusion: it sort of expands its context and prompt by association. It has no sense of direction of its own maybe. Once you know what you are doing, you can ride it fine. If not, it makes a misstep so later things go haywire, and you are left guessing why. (And how can you guess if you have not had the experience. We all grew with Commodores and suchlike. I recall Spectrums, Robotrons, and Poisks. My friend sits on exam committees, says the youth arrives flatlined after 3 years of GPT. His words. Whatever.)

> It has no sense of direction of its own maybe. Once you know what you are doing, you can ride it fine. If not, it makes a misstep so later things go haywire, and you are left guessing why.

This is a perfect description. Last week I asked claude opus to get AAC audio working in davinci resolve on linux. It managed to add aac in mov and mp4 containers very quickly. But mkv was another matter. For mkv files, resolve doesn't use ffmpeg. Instead, it has its own parser. Claude got totally lost down a weird rabbit hole trying to add aac support to resolve's mkv code. It was really struggling. Claude even knew it was lost - it kept telling me we should cut our losses and I should just release aac support without mkv.

Eventually I gave it the executable for davinci resolve on mac, which has aac support. Claude found the corresponding part of the code for the mac version and used it as a reference. Turns out, claude had made some much earlier mistake. Just like you said, it was going down a wrong path. Then it couldn't stop itself, and it kept making it worse.

Using the mac version of the binary as a reference, claude figured out how to get everything working very quickly. But - I'm left wondering. Maybe my real mistake was using Opus and not Fable. I wonder if fable would have been smart enough to figure out the mistake and course-correct.

Like Aladdin's jinni is a slave of the lamp, LLM is a slave of its context window :)

Small OT:

> Maybe I should document what I did!

Please do!

Me and a bunch of friends worked on a project that used diamond-types as the backing CRDT engine. It does indeed go brrrr. But damn, it took way too long to reconstruct what it was doing (we needed some more fine grained knobs, so we were playing with the frontier directly). We eventually moved out to something a bit better documented, which was a real pity. I really liked the general architecture and simplicity (of the text-only based version at least).

Thanks! If I can ask - where did you get stuck?

We were encountering a panic in the lib and could not really figure out why it was firing. The project kind of stalled due to lack of time and funding anyway unfortunately.

I think you nailed it. And that would make sense and should be expected. 99.9% of the world's software (the training data) is several orders of magnitude away from max performance.

Another possible confusing thing for an LLM is that getting close to max doesn't necessarily require any "tricks." A big part of getting in the ballpark is just not doing anything you don't have to. If program A is faster than program B, most of the time is not some magic algorithm. It's that program A just did less stuff.

> A stochastic search process with an executable optimization objective over space of programs S can only maintain or improve the objective

Reasons this doesn't follow: (1) Benchmarks never match real world use, and many optimizations the improve benchmarks degrade cases that aren't measured (think about how CPU cache behavior can be surprising) (2) In software performance optimization, frequently there is significant noise, from many sources. This makes it difficult to guarantee that a measured change is actually an improvement.

Both of those things are indicators of deficiencies in the testing process.

Reply on news.ycombinator.com