[dead]
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.
https://arxiv.org/pdf/2311.09205
In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.
But the idea they’d be better than a bigger model is cope, you’re pretty much always better off running the biggest one you can bring to bear within your constraints.
It feels like claiming that because a current generation laptop CPU is slower than a current generation HEDT CPU, we should ignore that the laptop CPU is faster and more efficient than the previous generation HEDT CPU.
Of course a huge system could do more, but these systems are constantly evolving and just in this month we've seen comparatively smaller models approach capabilities that were frontier AI less than a year ago. What you're saying would be true when we eventually find the limits of how much models can compress knowledge. Until then, all I think you're saying is that companies that are willing to burn more money can afford to ignore efficiency.
There's a million ways to define "better at writing prose", and it's only better at some of them. It's worse at many others.
Try using a LLM model for RAG embeddings and get back to us on that.
"Source: rare book ingested and shredded by Anthropic. No, you can't look it up and we can't show you the scan. The remaining open market copy is $5000. Trust me."