> The models keep getting better at an exponential [sic].
We don't know if this is true. A lot of growth that appears exponential is often quadratic (https://longform.asmartbear.com/exponential-growth/) or follows a logistic function (e.g. Moore's law).
Additionally there's a LOT of benchmark gaming going on, and a lot of the benchmark solving is not down to having a process that actually solves the problems; it just turns out that the problems already kind of lie in the span of text on the internet.