Hmm we may be talking of two different interpretations of intuition here. I agree that LLM weights contain representations of abstract concepts, as a lot of prior research has shown. This surely includes Mathematical concepts.

But to me that is analogous to what human brains do, and a bit different from intuition. I think of intuition as “heuristics”, typically developed through experience, that may link seemingly unrelated concepts via vague, hard-to-define associations, but which let us make mental leaps (or shortcuts) while reasoning. (Maybe analogous to System 1 / 2 thinking.)

On the other hand, LLMs can do both: build “intuition” from patterns in data AND brute force a huge amount of potentially unrelated concepts. This gets fuzzier when we realize that even these “concepts” themselves are gleaned from patterns in data! But my point is we necessarily have to take shortcuts to scale, whereas machines can scale with hardware.

This is of course a layman theory! But it could explain why these models are progressing so fast.