The upcoming GPT-3 moment for RL
mechanize.work
mechanize.work
The core claim that massive-scale RL will unlock generalization doesn't seem that surprising since we've seen the scaling hypothesis play out across ML. But "replication training" on software is interesting: learning by copying existing programs potentially unlocks a ton of complex training data with objective evaluation criteria.
To me, the big unanswered question is whether skills learned from replicating software would generalize to other reasoning tasks. That's a significant "if" - great if it works, pointless if it doesn't.
It would "work" but I think it will need even more scrutiny by experts to confirm what's correct and what needs to be re-generated. Please please no vibe accounting.
Funny you mention; There are multiple companies in Sweden working on AI/ML based accounting. It's not so different from AI/ML based automated driving.
I guess it's like automated driving a few years back - monitored by humans able to take over control. Step by step it'll become better and better until good enough for handle certain chosen areas.
The primary task in book-keeping is to classify transactions (from expense vouchers, bank transactions, sales and purchase invoices and so on) and slot them into the Chart of Accounts of the business.
LLMs can already do this well without any domain/business specific context. For example - a fuel entry is so obvious that they can match it into a similar sounding account in the CoA.
And for others where human discretion is required, we can add a line of instruction in the prompt, and that classification is permanently encoded. A large chunk of these kind of entries are repetitive in nature, and so each such custom instruction is a long-term automation.
You might have not been speaking about simple book-keeping. If so, I'm curious to learn.
Requirements for a human in the loop devolve to ticking a box by someone who doesn't realise the responsibility they have been burdened with.
Mark my words, some unfortunate soul will be then be thrown under the bus once a major scandal arises from such use of LLMs.
As an example, companies aren't supposed to use AI for hiring, they are supposed to have all decisions made by a human-in-the-loop. Inevitably this just means presenting a massive grid of outcomes to someone who never actually goes against the choices of the machine.
The more junior the employee, the "better". They won't challenge the system, and they won't realise the liability they're setting themselves up with, and the company will more easily shove them under the proverbial bus if there ever is an issue.
Hiring is too nebulous, too hard to get concrete data for, and too hard to inspect outcomes to properly check.
Financial auditing however is the opposite of that. It's hard numbers. Inevitably when discrepancies arise, people run around chasing other people to get all their numbers close enough to something that makes sense. There's enough human wiggle-room to get away with chaotic processes that still demand accountability.
This is possibly the worst place you could put LLMs, if you care about actual outcomes:
1. Mistakes aren't going to get noticed.
2. If they are noticed, people aren't going to be empowered to actually challenge them, especially once they're used to the LLM doing the work.
3. People will be held responsible for the LLM's mistakes, despite pressure (And the general sense of time-pressure in audit is already immense ) to sign-off.
4. It's a black-box, so any faults cannot be easily diagnosed, the best you can do is try to re-prompt in a way that doesn't happen.
The vague part is whether this will generalize to other non software domains.
A much harder task than writing said software
OK, but then you have to produce the detailed specification, working backward from the reference implementation. This is extremely non-trivial and it significantly weakens the TFA's parallels to pre-training, in which you don't need really need inputs other than raw text corpora.
I'm not saying this eliminates the idea outright, but I do think it hobbles it badly.
And you can use a fuzzer to augument that.
I’d propose the following architecture:
Step 1: Microsoft phi style - read code and write specifications using a frontier model. You could use an ensemble here to nitpick the spec; it’s only going to get written once. We also have of course many many rfcs and codebases that conform to them or where they do not we have an existing repository of bug reports, patches, forum complaints, etc.
Step 2-4: implement multilayer evaluation: does it compile? Does an existing model think the code complies with the spec on inspection? When it’s run on qemu are the key evals the same as the original software?
I propose most of steps 2-4 are automatable and rely on existing tooling and provide a framework that is, if not cheap, achievable. I’m also sure someone could improve this plan with a few more minutes of thought.
To me the interesting question is - will this add capabilities at current model sizes? My prior is yes in that the current behemoth size models feel like they are only incrementally better than 1/10 size distills. I interpret that to mean we haven’t gotten the most out of these larger scales. I will note Dario disagrees on this - he’s publicly said we need at least 10x more scale than we have now.
Imagine a long conversation. It is hard to judge if an AI response was useful or not immediately, but if you know the following 20 messages, it might be easy to infer. Not only you can see how it went, but sometimes you get real world validation.
For example a user comes to a LLM with a task, takes an idea, tries it in reality. Later they return, maybe in a new chat session, and continue iterating. You get real world testing of LLM responses through people.
This can be used to generate "preference scores", and train a preference model, with which you can do RLHF. So the user privacy is protected.
I call this the human-AI experience flywheel. Of course the larger the user base, the more experience the model collects. At the moment OpenAI has 500M users, they probably generate 0.5T interactive tokens/day. Those tokens go both into human brains and LLM logs.
It’s not about environment engineering anymore, it's about consequence harvesting. Meaningful validation emerges from systems actually being used by humans for real purposes.
For a given tax question, could you come up with the same memo quoting the same sources and same conclusion?
This article suggests scaling up RL by exposing models to thousands of environments
I think we can already achieve something similar by chaining multiple agents:
1. A “requirement” agent that uses browser tools to craft detailed specs from docs.
2. A coding agent that sets up environments (Docker, build tools) via browser or CLI.
3. A testing agent that validates code against specs, again through tooling.
4. A feedback loop where the tester guides the coder based on results.
Put together, this system becomes a fully autonomous development pipeline-especially for small projects. In practice, I’ve left my machine running overnight, and these agents propose new features, implement them, run tests, and push to repo once they pass. It works surprisingly well.
The main barrier is cost—spinning up many powerful models is expensive. But on a modest scale, this method is remarkably effective.
what
99% of the time their reasoning is laughable. Or even if their reasoning is on the right track, they often just ignore it in the final answer, and do the stupid thing anyway.
The result is a 2x2 matrix where several quadrants are deeply concerning to me.
I give you a 2x2x2 matrix.
That'd be a tensor, no?
I fully believe that LLMs encode enormous amounts of knowledge (some of which is even correct, and much of which their operator does not personally possess), are capable of working quickly and ingesting large amounts of data and working quickly, and have essentially no judgment or particularly strong intelligence of the non-memorized sort. This can still be very valuable!
Maybe this will change over the next few years, and maybe it won’t. I’m not at all convinced that scraping the bottom of the barrel for more billions and trillions of low-quality training tokens will help much.
It's not even a different strategy. It's just using rhetoric in a more limited way, and without human emotion.
These are style over substance machines. Their cognitive abilities are extremely ragged and unreliable - sometimes brilliant, sometimes useless, sometimes wrong.
But we give them the benefit of the doubt because they hide behind grammatically correct sentences that appear to make sense, and we're primed to assume that language = sentience = intelligence.
I think the problem is that despite feeding it all the context and having all the right MCPs agents hooked up, is that there isn't a human-in-loop. So it will just reason against itself causing these laughable stupid decisions. For simple boilerplate tasks this isn't a problem. But as soon as the scope is outside of a CRUD/boilerplate problem, the whole thing crumbles.
My personal experience the past five months has been very mixed. If I "let 'er rip" it's mostly junk I need to refactor or redo by micro-managing the AI. At the moment, at least for what I do, AI is like a fantastic calculator that speeds up your work, but where you still should be pushing the buttons.
I haven't seen an LLM stay on task anywhere near that long, like...ever. The only thing that works better left running overnight that has anything to do with ML, in my experience, is training.
I will assume you passed high-school based on your looks and not on your abilities.
99% chance you're using the wrong model.
Effective tool use is a valuable skill, arguably the only one that still matters.
I very much disagree. For the larger, more sophisticated stuff that runs our world, it is not cost that prohibits wide and deep automation. It's deeply sophisticated and constrained requirements, highly complex existing behaviors that may or may not be able to change, systems of people who don't always hold the information needed, usually wildly out of date internal docs that describe the system or even how to develop for it, and so on.
Agents are nowhere near capable of replacing this, and even if they were, they'd change it differently in ways that are often undesirable or illegal. I get that there's this fascination with "imagine if it were good enough to..." but it's not, and the systems AI must exist in are both vast and highly difficult to navigate.
I'd argue the opposite of your stance: we've never had a chance at a fresh start without destruction, but agents (or their near-future offspring) can hold our entire systems "in nemory", and therefore might be our only chance at a redo without literally killing ourselves to get there.
Additionally, I disagree with your point:
> The status quo system you describe isn't objectively optimal.
On the basis that I would challenge you or anyone to judge what is objectively optimal. Google Search is a wildly complex system, an iceberg or rules on top of rules specifically because it is a digital infrastructure surrounding an organic system filled with a diverse group of people with ever-changing preferences and behaviors. What, exactly, would be optimal here?
I don't know where you're going with discussion of destruction and killing, but even fairly simple consumer products have any number of edge cases that initial specifications rarely capture. I'm not sure what "objectively optimal" is supposed to mean here, either.
If a spec described every edge case it would basically be executable already.
The pain of developing software at scale is that you're creating the blueprint on the fly from high-level vague directions.
Something trivial that nevertheless often results in meetings and debate in the development world:
Spec requirement 1: "Give new users a 10% discount, but only if they haven't purchased in the last year."
Spec requirement 2, a year later: "Now offer a second product the user can purchase."
Does the 10% discount apply to the second product too? Do you get the 10% discount on the second product if you purchased the first product in the last year, or does a purchase on any product consume the discount eligibility? What if the prices are very different and customers would be pissed off if a $1 discount on the cheaper product (which didn't meet their needs in the end) prevented them from getting a 10$ discount 9 months later (which they think will)? What if the second product is a superset of the first product? What if there are different relevant laws in different jurisdictions where you're selling your product?
Agents aren't going to figure out the intent of the company's principal's automatically here because the decision maker doesn't actually even realize it's a question until the implementers get into the weeds.
A sufficiently advanced agent would present all the options to the person running the task, and then the humans could decide. But then you've slowed things back down the pace of the human decision makers.
The complexities only increase as the product grows. And once you get into distributed or concurrent systems even most of our code today is ambiguous enough about intent that bugs are common.
Yes this resonates completely. I think many are forgetting the purpose of formal language and code was because natural language has such high ambiguity that it doesn't capture complex behavior
LLMs are great at interpolating between implicit and unsaid requirements but whether their interpolation matches your mental model is a dice throw
Imagine if...nevermind.
So...like SAP then?
What I don't necessarily see is the generalization factor - say, we improve software engineering and math performance through RL learning (probably easier for software engineering than math due to available training corpus). If that generalization factor doesn't hold, due the economics still work out? An expert-level software model would be useful to our profession, sure, but would it be enough to recoup the training costs if it's not applicable to other industries?
For a complete piece of software the answer might be 10 million tokens, and that doesn’t even count the reasoning.
Now imagine that there was a mistake at some point. The model will need to go back to fix it, and understand the cascade of things the bugfix changed. It might be possible to keep that all in the context window but that seems like it won’t scale.
Currently, an LLM with everything under the sun in the context window behaves rather poorly and gets confused by that, even if we're not exceeding the context window length. Although it'd be certainly also interesting to train for increasing the maximum _actually_ usable context window length, I don't know how feasible that would be.
> How to Scale RL to 10^26 FLOPs (blog.jxmo.io/p/how-to-scale-rl-to-1026-flops)
The basic premise behind both essays is that for AI to make another big jump in capabilities, we need to find new data to train on.
My proposal was reusing text from the Internet and doing RL on next-token prediction. The linked post here instead suggests doing 'replication training', which they define as "tasking AIs with duplicating existing software products, or specific features within them".
It sounds cool as marketing. It helps improve LLMs a bit. And it will never yield s.th. like an AGI or anything that is "reasoning". Unless you also redefine the word reasoning of course.
1) If only there was a cryptocurrency tied to training AI models and make crypto grinding useful than maths that solve no real-world problem external to the token creation itself.
2) Larger and larger AI models, you start to get more hallucinations, maybe we should focus on dedicated highly tuned models for dedicated aspects and have a higher up conductor model that knows what to farm out to which models and from there combine and send out further requests to other models etc to come to a result. Certainly, the need for highly tuned niche models, after all, language recognition as an example, a model that could identify the language, local dialect and accent, that would then use a language model tuned better for that speaker it is recognising. That approach feels like the way over one large model that does it all itself.
I have sadly lost access to my wallet since.
So your plan is to train a MLP to black box replicate complex and highly non-linear encryption algorithms through gradient descent?
LLMs can be trained on an unsupervised way on static documents. That is really the key feature that lets them be as smart and effective as they are. If you had every other technology that LLMs are built on, and you didn't have hundreds of terabytes of text laying around, there would be no practical way to make them even a tiny tiny fraction as effective as they are currently.
RL != only online learning.
There's a ton of research on offline and imitation-based RL where the training data isn't tied to an agents past policy - which is exactly what this article is pointing to.
From my understanding, RL is a tuning approach on LLMs, so the outcome is still the same kind of beast, albeit with a different parameter set.
So empirically, I actually thought that the lead companies would already be strongly focused on improving coding capabilities, since this is where LLMs are very effective, and where they have huge cashflows from token consumptions.
So, either the motivation isn't there, or they're already doing something like that, or they know it's not as effective as the approaches they already have.
I wonder which one it is.
What you're referring to is actually just one application of RL (RLHF). RL itself is much more than that
* reading and understanding long, complicated, detailed instructions,
* executing those instructions meticulously and precisely, without errors,
* noticing its mistakes, if there are any along the way, and recovering from them,
* not settling prematurely for solutions that look "good enough" but aren't, and
* undertaking large, complicated projects which previously could be completed only by teams of human experts.
There's a good chance the OP is right, in my view.
We sure live in interesting times!
>Each replication task consists of a detailed specification and a reference implementation. The central idea is that AI models are trained to produce an implementation that precisely matches the reference behavior.
I really don't see the connection from the statements in the article's content, and the assertion near the start that:
>Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks.
There's no clear reason outlined in the piece to describe why narrow & well-scoped 1-person-day tasks might scale up to 10,000-person-year projects. If they did, we should expect far more 10,000-person-year projects in the real economy, because the learning curve for firms scaling would be something approximating a straight line. There are very few 10,000-person-year projects, and very many 1-person-day projects.
It seems more like this will spend an unimaginable amount of compute, in order to produce models which are incredibly good at a very precise form of IP theft, and not especially good at any generalisable skills. It's so ludicrously rare that an engineer (or author, illustrator, etc) is tasked with "create a pixel-perfect reimplementation of this existing tool".
A smell big success? Copyright laundering is the killer app of AI this far.
It's very hard to define (in way to create lints) what makes core readable and maintainable. Using other LLM for this task could cause original model to game the system by abusing some weaknesses in the other model.
for other tasks, how do you even evaluate thinks like eg user experience/app design? how to properly evaluate pelican ridding bicycle?
These kind of "rookie mistakes" are not things that any modern LLM is likely to do. Indeed, I had to argue quite strongly with Gemini recently when I was learning a new tool (so basically just playing around with a fully local setup) and I hardcoded an API key then tried to commit it. The LLM did NOT like that! I had to carefully explain that this was a toy repo.
The argument against this (by Gemini) was that toy repos often grow into production tools so it's best to follow basic security rules from the start. Which, to be fair, is a good argument. I still committed the key though (and deleted the repo a day or so later).
One network tries to clean-room implement the hash function. The other network tries to find an input for which the reference implementation behaves differently.
Non rigorous reasoning is at the root of the problem here. AI hype is often indistinguishable fro AI slop, because those who believe it are also not very good at demanding and asserting rigor.
This is a great hypothesis for you to prove one way or the other.
> Doing this effectively will produce RL models with strong few-shot, task-agnostic abilities capable of quickly adapting to entirely new tasks.
I am not sure if I buy that, frankly. Even if you were to develop radically efficient means to create "effective and comprehensive" test suites that power replication training, it is not at all a given that it will translate to entirely new tasks. Yes, there is the bitter lesson and all that but we don't know if this is _the_ right hill to climb. Again, at best, this is a hypothesis.
> But achieving this will require training environments at a scale and diversity that dwarf anything currently available.
Yes. You should try it. Let us know if it works. All the best!
Step 2. Train the RL network. In the mean time drink coffee or work on plan of world domination.