We've had that concept for quite a long time now, in the form of Lora [1] and similar fine-tuning techniques.
It first got popular for StableDiffusion to teach the image generation models new concepts.
We could easily live in a world where you can train / build Loras to encompass your entire code base history, company knowledge base, new skills, etc.
Then the models would start with a baseline that already has all the important knowledge without needing to cram it into the context.
This still isn't on the fly learning, but you could imagine daily or weekly training runs to regularly incorporate new knowledge.
I think the main reason this hasn't happened yet is that the shared batch based efficient serving architectures used today wouldn't support that structure well.