> I could never imagine an LLM in its current form inventing something as elegant as the Fourier transform.

This is the crux imo. It doesn't really matter what absolute capability AI has at the moment, but whether we are on track with respect to architecture and training approaches. In my mind the only test for AGI is, if the thing trained up to the cut-off of Fourier's time (or Descartes, Newton etc), it should arrive at or exceed their insight.

^ As a measure of intelligence. But it's probably the case either way that LLM is more valuable in terms of the coordinated grunt work we'll put it to, than leaps of insight.