Hacker News Reader

Comment by dprkh | Hacker News Reader

I used an LLM to build this

It has good test coverage, mostly unit tests but also a number of end-to-end tests. I also made the LLM build a benchmark, which you can find at the bottom of the readme. It is obviously slow, but I thought that it is good enough to work. When I tried to write a 1 GiB file, I found that it broke down, and after writing half the file, the speed went to under one megabyte per second. Implementation is 10k+ LoC, and I have no idea what is going on there.

View on HN

That's interesting because I would feed that benchmark back into the agent and loop over it, to see how much faster you could get it, and agents are really good at that kind of recursive optimization. And I would definitely add at least a simulated 1GiB write test, probably a real one honestly, if I was building something like that.

At least with agent-run tests I care about loop speed a lot, but I care about complete coverage more, so having the odd heavy weight full stack integration test is fine, I think.

You're right. This was just a performance issue, but what if next time it is a corruption bug or a security vulnerability or really anything that can cause real consequences if happened in production? I don't think that LLM systems are inherently bound to have this flaw, but I think that we are pretty far from harnesses and algorithms becoming advanced enough so that the LLM system can kind of continuously evaluate its output and ensure it is good in all aspects.

I don't know about that, Fable is, when properly guided, a better engineer for those things than I am. Narrow breadth, weird priorities, myopic and ivory tower as hell, but superhuman. Maybe that says more about me, or maybe not, but certainly it's caught bugs I would not have, and point it at things like a fuzzer, woo buddy, it has been many years since I broke out valgrind and nailed down a memory leak, but it sure can.

Yep.

Reply on news.ycombinator.com