Comment by btown | Hacker News Reader

Comment by btown | Hacker News Reader

Parent Full thread

> This was a clean-room implementation (Claude did not have internet access at any point during its development); it depends only on the Rust standard library. The 100,000-line compiler can build Linux 6.9 on x86, ARM, and RISC-V. It can also compile QEMU, FFmpeg, SQlite, postgres, redis, and has a 99% pass rate on most compiler test suites including the GCC torture test suite. It also passes the developer's ultimate litmus test: it can compile and run Doom.

This is incredible!

But it also speaks to the limitations of these systems: while these agentic systems can do amazing things when automatically-evaluable, robust test suites exist... you hit diminishing returns when you, as a human orchestrator of agentic systems, are making business decisions as fast as the AI can bring them to your attention. And that assumes the AI isn't just making business assumptions with the same lack of context, compounded with motivation to seem self-reliant, that a non-goal-aligned human contractor would have.

Interesting how the concept of a clean room implementation changes when the agent has been trained on the entire internet already

To the best of my knowledge, there's no Rust-based compiler that comes anywhere close to 99% on the GCC torture test suite, or able to compile Doom. So even if it saw the internals of GCC and a lot of other compilers, the ability to recreate this step-by-step in Rust is extremely impressive to me.

The impressiveness of converting C to Rust by any means is kind of contingent on how much unnecessary unsafe there is in the end result though.

None - all references to 'unsafe' are in comments about the codegen: https://github.com/search?q=repo%3Aanthropics%2Fclaudes-c-co...

I think the careful response to this is:

(1) There are compilers written in C in the training set

(2) LLMs demonstrably can near-perfectly memorize training-set inputs (see other comments here)

(3) LLMs are very good at translation tasks (natural language or code, e.g.: C to Rust)

I don't think this necessarily completely deflates the impressiveness of this accomplishment, but it does qualify it to some degree.

You can use ai coding tools to create test suites, specifications, documentation, etc. And you can use them to scrutinize those, review them, criticize them, etc. Not having a test suite just means you start with creating one. Then the next question of course becomes "for what?".

This indeed puts human prompters in a position where their job is to set the goals, outline the vision, ask for the right things, ask critical questions, and to correct where needed.

Human contractors are a good analogy. Because they tend to come in without too much context into a new job. Their context is mainly what they've done before. But it takes time to get up to speed with whatever the customer is asking for and their context. People are slightly better at getting information out of other people. AI coding tools don't ask enough critical questions, yet. But that sounds fixable. The breakthroughs here are as much in the feedback loops and plumbing around the models as they are in the models themselves. It's all about getting the right information in and out of the context.

You would spend years verifying the tests actually work where as the tests for this accomplishment were already verified by humans over decades

Agreed, but the next step is of having an AI agent actually run the business and be able to get the business context it needs as a human would. Obviously we're not quite there, but with the rapid progress on benchmarks like Vending-Bench [0], and especially with this teams approach, it doesn't seem far fetched anymore.

As a particular near-term step, I imagine that it won't be long before we see a SaaS company using an AI product manager, which can spawn agents to directly interview users as they utilize the app, independently propose and (after getting approval) run small product experiments, and come up with validated recommendations for changing the product roadmap. I still remember Tay, and wouldn't give something like that the keys to the kingdom any time soon, but as long as there's a human decision maker at the end, I think that the tech is already here.

[0] https://andonlabs.com/evals/vending-bench-2

Reply on news.ycombinator.com