According to the Latex source, one of the original titles for this paper was "First Contact With an AGI System".
What is the "hard line" that makes something AGI or not AGI? Because IMO it looks like GPT4 is somewhat AGI, but also the older models possibly all the way down to even Markov chains: it's just that this AGI is nowhere near human-level.
However, I think we can certainly prove when it doesn't. For instance, the fact we need an external plugin to get the model itself to return text claiming 1+1=2, tells me that GPT4 cannot reason about numbers in the abstract, and therefore lacks abstraction ability.
I think we're so strongly biased against a deeply uncomfortable reality that there may not be a hard line, we don't even want to consider the alternative.
But to make matters worse, or more muddied, 1+1=1 can be a valid mathematical statement. It simply depends upon the set and group you have, or if you're doing modular arithmetic, etc. Sometimes you're given a unital magma. So, there's still a heavy dependency on context for the problem setup, but the underlying discrete and deterministic rules that are applied to the context is less malleable than other context switches in NLP LLMs do well in (such as language styling).
Humans certainly have context windows. Try asking your CEO about some lines of code in your work. Humans have a fairly large one, I give you that and it is fuzzy.
An example about ping pong players. The pace of their movements is too fast for conscious thinking, so it's all trained reflexes with some overall strategic planning trying to keep up with events. There is no time to think about anything. Is intelligence suspended there, at least the general one? Then the same person stops playing and gives an interview about the game and full general intelligence turns on again.
Human level performance on general tasks (you can't be just good in 90%, you should avoid being terrible in the remaining 10%).
Though LLMs don't need to be AGI, to have a catastrophic economic impact.
It's pretty intelligent and rather general too, so at least by the definition that doesn't include mandatory consciousness it would mostly fit. And consciousness is pointless for a robotic system because it doesn't add anything practically useful. Just because agency has to be provided by the user doesn't make it any less of an AGI I'd say.