How would we do that? There's no historic precedent for that. Its fundamentally an information theory thing: what's the max amount of intelligence you can get out of 1 KB/MB/GB? There has to be a limit and I'm not convinced that it's far off.
If we are talking about all knowledge, then I agree that the compression ratio is very impressive already.
Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.
Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.
Edit: I guess at the moment this is just having an LRU cache of experts
See Karpathy's "Cognitive Core" idea. (I don't have a good link)
Standard Def TV was plenty for 50 years. But when more was on offer, everyone went for it, and now you can't even buy a 480p TV.
2K, 4K, and 8K+ TVs have been around forever, and although 4K has become the norm, it's widely accepted that there isn't much benefit for most people above 1080p
Is it? By whom? For larger TVs, closer distance, or a combination of the two, there absolutely is value from 4k. https://i.rtings.com/images/optimal-viewing-distance-televis...
Though I'm unlikely to ever upgrade from 1440p on my computer, personally.
The 1080p thing is true in the age of 40” TVs but not so much now, think Costco has a 95” TV if not 100”.
Personally I think 4k (with HDR) is good enough for consumer use, so agree with the parent theoretically just not quantitatively.
It did take around 20 years from DVD to 4k Bluray though.
And when (not if) we get to the 4k (or 8k) equivalent of LLMs, they'll just be baked into hardware and we'll have near instant responses while running locally.
There is no future where OpenAI, Anthropic, etc survive with their current business model; at some point we will hit a point where training a new model is done only every 5 years or so, and in that scenario we aren't going to be running models of pricey server GPUs with latencies measured in seconds and full responses measured in minutes.
We'll be running locally with sub-millisecond latencies and responses measured in milliseconds. There is no way for any big company to compete with the current business plan of selling inference or subscriptions.
> a point where training a new model is done only every 5 years or so
In a way it is actually the same thing that makes us accept AI as working in the first place. It only needs to be good enough for human perception. The same is probably true for compute.
Plus there is a chance the actual scaling that matters is beyond our reach. Think instead of TB models, PB or ZB models. We don't even have that kind of information. Humanity in its entire history hasn't generated 1ZB of information.
On a task that corresponds to the benchmarks, yes absolutely.
The very definition of diminishing returns.
And yet when we hit 4k, that's were people just stopped buying higher res. 8K is still useful, but only when the screen is so large that it doesn't fit in the room :-/
There must be a limit, I agree, but there have been no signs of approaching it yet. The most recent cohort of small models have shown the biggest leap in capability so far.
Some groups are baking models into silicone, Deepmind has an example, it gets 18,000 tokens/sec on Llama 3.1, not sure about parameter size
While some other groups are baking silicone into models :)
but yes misplaced e
Maybe? Human science history is absolutely littered with examples of things that were “constrained” by a fundamental law … until they weren’t, because we’d misunderstood or misapplied the law.
LLMs haven’t been on the scene for very long. There’s still lots of room for efficiency discoveries. Plus, I keep hearing quantum is going to be a big deal in the next few years.
The question then is, how much memorization is really needed for intelligence if you can query structured information?
Sure, it might be very near using the current approach, but... it might also be might be very far off because we are using the wrong approach.
I mean, look at the max amount of intelligence you can get out of a human brain powered by two bananas...