Hacker News Reader

>METR measures the longest software engineering task a frontier model can complete successfully half the time

Longest? Yes, they actually use actual wall clock time. I don't think this metric makes any sense. I don't know about the others, I stopped reading at this point.

View on HN

The others are an IQ test (TrackingAI), a test of Ph.D. questions across multiple domains (Humanity's Last Exam), and graphical pattern matching (ARC-AGI-2).

What's interesting is that while they are rather different in nature (yes it is odd that METR measures clock time as opposed to iterations etc.) but the behavior of the resulting improvement curves are extremely close.

That's the punchline: 4 independent measures point to the same conclusion. That increases the chance that the conclusion is correct.

Reply on news.ycombinator.com