At some point we have to be running into some inherent mathematical limits of knowledge compression, right? No way the knowledge benchmarks on these 8B models will keep getting better without overfitting on these benchmarks
As an example for a similar approach, Teapot AI has trained very small models https://teapotai.com/models to only answer questions where the answer can be found within the context window, and although not perfect, they do quite well at this compared to larger, more general models.
Humans train "thinking methodology" first and then know how to use it while accessing data and to build knowledge.
Humans do not memorize at once all text in existence, that's totally stupid.
Already thinking humans specialize in disciplines: math, chemistry, IT, cooking, etc while still using new data.
All of that computing is local- on the LAN of the brain.
So if some "agents" wants to help then there is zero need for computation outside of home/corporation/car local area network.
Licenses ??