Chollet writes he expects AGI now sooner than 2030, "given progress is happening faster than I expected."
The RL phase is the most similar mechanism I know of that comes to my mind, but I'm not sure it could be adapted to fill that gap.
My totally unsubstantiated theory is that this is the missing link towards what most humans would consider AGI. I don't see this as intractable, but it may require some substantial change in architecture.
It first got popular for StableDiffusion to teach the image generation models new concepts.
We could easily live in a world where you can train / build Loras to encompass your entire code base history, company knowledge base, new skills, etc.
Then the models would start with a baseline that already has all the important knowledge without needing to cram it into the context.
This still isn't on the fly learning, but you could imagine daily or weekly training runs to regularly incorporate new knowledge.
I think the main reason this hasn't happened yet is that the shared batch based efficient serving architectures used today wouldn't support that structure well.
For sure, doing it for all users would be economically unfeasible. I wonder if the labs are experimenting with something similar, though.
In some sense, creating good new abstractions externally and learning those is a form of "learning", on a very large timescale. The AI model is not necessarily doing the whole "look at the whole space holsitically and find a key invariant", but if you let other people do that you can enable new capabilities that were previously unknown.
Tools and capacities, man.
Some of the linear RNN layers in recent models are provably doing SGD in hidden space during inference
1. Is this learning persistent?
2. Do they verify these new lessons against core principles?
3. Do they and protect themselves/ignore requests if these new lessons contradict those core principles?
Humans do that from the time they're 3 years old (not that well, but they do do it).
So the next step is to ask for evidence and ideally independent and peer reviewed research.
And ICL dates all the way back to 2020, at least: https://arxiv.org/abs/2005.14165
My guess is that we'll just ignore it and make money along the way and every 2-3 months we'll have the equivalent to "Equifax gets hacked and millions of user records are stolen", etc. (this time with the LLM itself doing the hacking at someone's behest - accidental or not).