Anyone else experimenting with letting LLMs generate their own or sub-agent system prompts?
Instead of hundreds of thousands of us downloading the latest and greatest model that won't fundamentally update one bit until we're graced with the next one, I would think we should all be able to fine-tune the weights so that it can naturally memorize new additional info and preferences without using up context length.
In fact, systems should be LLM agnostic or use different models for different needs.
I don't believe building something into the model will ever be the solution though. It is interesting what Google is trying to do with model caching but at the end of the day I believe the strength of agents here will rely heavily upon modularity.