Forecaster reacts: METR's bombshell paper about AI acceleration

I haven't yet seen such a graph set side by side with hyperscaler capex, including forecasts. From a safety perspective the two should be correlated and we can easily say where the growth stops to be exponential (can't double capex every 3 months!)

If performance decouples from capex (I expect multiple 'deepseek moments' in the future) it's time to be properly afraid.

Before I read the article, I like to know who wrote it to understand whether I should spend or waste my time reading the article.

After a bit of digging it turned out the author isn't lying. There actually is a contest for forecasters and of those who participated, he was consistently in relatively high positions. See 2022:

https://www.lesswrong.com/posts/gS8Jmcfoa9FAh92YK/crosspost-...

At the same time, he is the CEO of Rethin Priorities think tank which seems to be excellent work for the society at large:

https://rethinkpriorities.org/

So I'm bookmarking this article and will read it during my reading session today.

This is comparable to another research/estimate with similar findings at https://ai-2027.com/ I find the proposed timelines aggressive (~AGI in ~3 years), but the people behind this thinking are exceptionally thoughtful and well-versed in all related fields.

I hear these superforecasters are really good at predicting what happens in the next ten minutes

Task:

Create an imahe of an analog clock that show 09:30 a.m

Last time I checked ChatGPT failed miserably, took my 10 year old nephew a minute.

Maybe it's bad to extrapolate those trends beacuse there is no constant growth. How looked the same graph when self driving took off and how is it now?

OK, I see one glaring problem with this approach (having used both Claude 3.7 and o3). When they talk about 50% reliability, there is a hidden cost: you cannot know before hand whether the response (or a series of it) is leading you to the actual, optimal or good enough solution, or towards a blind alley (where the solution doesn't work at all, works terribly, or, worst at all, works only for the cases tested). This is more or less clear after you check the solution, but not before.

So, because most engineering tasks I'm dealing with are quite complex and would require multiple prompts, there is always the cost of taking into account the fact that it will go bollocks at some point. Frankly, most of them do at some point. It's not evident for simple tasks, but for more complex they simply start inserting their BS in spite of often excellent start. But you are already "80% done". What do you do? Start from scratch with a different approach? Everybody has their own strategies (starting a new thread with the contents generated so far etc) but there's always a human cost associated.

"Forecasters" are grifters preying on naive business types who are somehow unaware that an exponential and the bottom half of a sigmoid look very much like one another.

As usual with these bombastic blog posts, you have to follow the paper trail to find the caveats that deflate the whole thing.

From the papers about the tasks used in this estimation, we can easily find out that:

LLMs never exceed 20% success rate on tasks taking more than 4h:

HCAST: Human-Calibrated Autonomy Software Tasks https://metr.org/hcast.pdf

LLMs hit a score ceiling at around 1-2h mark; LLM performance drops off steeply with increase in LOC count, never breaching 50% success rate after 800 lines and below 40% at the longest one at ~1600 lines.

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts https://arxiv.org/pdf/2411.15114

And from the METR paper itself, only one model exceeds the 30 min "human task length" with 50% success, reaching that rate at 59 min "human task length", and at 80% no models but one can go over 4 min and one gets to 15 min "human task length" at that rate.

It goes on to talk about extrapolating from this SOTA 59 min "time horizon" to ~168h, arguing that this is about one month of full time work and models that could breach 50% success rates at that time span could be considered transformative because they "would necessarily exceed human performance both at tasks including writing large software applications or founding startups (clearly economically valuable), and including novel scientific discoveries."

Now despite the fact that "The mean messiness score amongst HCAST and RE-Bench tasks is 3.2/16. None of these tasks have a messiness score above 8/16. For comparison, a task like ’write a good research paper’ would score between 9/16 and 15/16, depending on the specifics of the task."

But according to the METR paper, on the 22 '50% most messy' >1h tasks no model even breaches 20% success rates. And that is for tasks above "messiness = 3.0" on a set of tasks that goes to 3.2 "messiness".

So there is absolutely no record of LLMs exceeding 50% success rates above 3.0 messiness ratings 1h long tasks but they are happy to claim that they see a trend towards 50% success rates at 168h long tasks approaching 9-15/16 messiness ratings? That's even assuming that their own estimate for the messiness of 'writing a good research paper' tops the chart is comparable to 'novel scientific discoveries', or 'writing large software applications or founding startups', which would seem to be many times messier than 'writing a good research paper', let alone doable in one month.

Measuring AI Ability to Complete Long Tasks https://arxiv.org/pdf/2503.14499

So let's talk about the blog post claims that are not backed by the contents in these papers:

"Claude 3.7 could complete tasks at the end of February 2025 that would take a professional software engineer about one hour."

This is incredibly misleading. Claude 3.7 is the only model that was able to achieve a 50% success rate on tasks that are estimated to take humans at least 1h to complete. We should note that the METR paper also shows that for the 50% "messier" tasks no model even breaches 20% success rate. It should be noted that the HCAST set of tasks has 189 tasks of which only 45 breach 1h baseline estimates. The METR paper uses a 'subset' of HCAST tasks but it is not clear which ones and what the baseline time cost estimates for these look like.

"o3 gives us a chance to test these projections. We can see this curve is still on trend, if not going faster than expected"

This was a rushed evaluation conducted on a set of tasks that is different from that in the original paper, making the comparison between datasets spurious.

Also, this seems relevant:

"For the HCAST tasks, the main resource constraint is in the form of a token budget which is set to a high enough number that we do not expect it to be a limiting factor to the agent’s performance. For our basic agent scaffold, this budget is 2 million tokens across input and output (including reasoning) tokens. For the token-hungry scaffolds used with o1 this budget is 8 million tokens. For o3 and o4-mini, which used the same scaffold as o1 but did not support generating multiple completions in a single request, this budget is 16 million tokens."

https://metr.github.io/autonomy-evals-guide/openai-o3-report...

Back to the blog post:

"We can actually attempt to use the METR paper to try to derive AGI timelines using the following equation:

days until AGI = log2({AGI task length} / {Starting task length}) * {doubling time}"

To find numbers to plug there, it makes some assumptions like "1h45 upper bound seems low let's boost by 1.5x" and then "but real world is messy so let's just divide the time by 10" which flies in the face of the fact that for tasks above messiness of 3/15 the models never breach 20% success rates. And considering the whole task set goes to 3.2/15 messiness score, that means that anything above that is a null data point. So this "Let’s assume true AGI tasks are 10x harder" assumption alone should drive "task length" to zero.

"Additionally, we likely need more than 50% reliability. Let’s assume we need 80% reliability, which adds a 4x penalty, plunging starting task length down further to 3min45sec."

At 80% reliability, only 2 models breach 4 minutes, with the best one approaching 15 minutes. And that is at messiness score of at most 3.2. But the best model regresses from the 2 bellow it on the overall estimation, and none breach even 20% reliability above that.

So none of the modelling is valid, but the sleight of hand is offering the AGI formula and the 168h task length value for 'AGI' which of course is spurious because we're than talking about an AGI that cannot do any task messier than a 3.2/15, and a doubling rate of 3 to 7 months, we are stuck with accepting AGI is at most 10 years away as the blog post claims. But at most we can expect models that can do a few hundred lines of code edition at a 50% reliability rate on projects that would take around a month. And the METR paper has it's own projection of a "1-month AI" (which it never claims to be AGI) with 50% chance between end of 2027 and early 2028. So let me know if you have any 1 month projects under 800 LOC that you may need, with a 50% chance to be able to use an LLM to have a 50% shot at eventually getting it right, in about 2-3 years!

I think the author is missing the point of why these forecasts put so much weight on software engineering skills. It’s not because it’s a good measure of AGI in itself, it’s because it directly impacts the pace of further AI research, which leads to runaway progress.

Claiming that the AI can’t even read a child’s drawing, for example, is therefore not super relevant to the timeline, unless you think it’s fundamentally never going to be possible.

> About the author: Peter Wildeford is a top forecaster, ranked top 1% every year since 2022.

What?

And by 2035, AI will be able to complete tasks that would take millennia for humans to complete.

https://xkcd.com/605/

> Does this mean that AI will be able to replace basically all human labor by the end of 2031?

Betteridge's Law: no.

Even the much more limited claim of AI replacing all white-collar keyboard-and-email jobs in that time frame looks questionable. And the capex is in trouble: https://www.reuters.com/technology/microsoft-pulls-back-more...

On the other hand, if that _does_ happen, what unemployment numbers are you expecting to see in that time period?

> This is important because I would guess that software engineering skills overestimate total progress on AGI because software engineering skills are easier to train than other skills. This is because they can be easily verified through automated testing so models can iterate quite quickly. This is very different from the real world, where tasks are messy and involve low feedback — areas that AI struggles on

Tell me you've never coded without telling me you've never coded.