I'm smarter than my dog, yet I am frequently unable to foresee my dog's actions.
Serious response: we have no idea what ASI means or might be like. It's literally sci fi (at the moment).
LLMs have a significant advantage in being trained on all attainable data of any sort about human behavior, which we've established quite a rich record of over the years.
That's true, which is my point: intelligence is not a scalar value. LLMs are intelligent from us in different ways, as a result of not being human, despite having access to a record of human behavior.
To bring this back to the dog analogy: we have a rich record of dog behavior and yet people still don't know how to be smarter than dogs in dog behavior. Because we're not dogs.
Yes.
Unfortunately, this is part of the problem.