...I'm just asking questions here... how sure are we of this?
If you'd asked me six years ago whether we could compress all of human knowledge into a 1 TB file, I would have said no, and yet here we are.
If you can do 1 TB, why not 30 GB? It's, like, within one order of magnitude.
On a more serious note, it depends on your cutoff for "entire human knowledge". It's easy to prove for a generous interpretations of "entire human knowledge" that it can't be done, but hard for something like "all useful human knowledge".
Moby Dick, uncompressed, is ~1MB. Compressed, it's around 500KB.
I feel fairly certain that one could fit all of the textual knowledge required to cultivate a world-class <insert name of preferred professional knowledge worker> in <60,000 Moby Dicks. (Arguably in <5,000 Moby Dicks with intense effort/pruning).
Or maybe not, maybe there's a world model needed for human level at any profession that is very hard to quantify and requires more than 30GB by itself.
I just think about all the content I’ve consumed in my life to become a professional software developer and I would be very surprised if it couldn’t be adequately represented by <30GB of uncompressed text. Most of the work was in “training”, not really in data.
The “foundational overlap” of K-12 education is identical for most professions with 2-8 years of “finishing” on top.
My mental model is that the budget is spread across maybe 20% genetics (most of our instinctive/genetic information is surely pretty useless), 50% k-12 education, 30% for professionally-specific knowledge.
I think the problem is that the conversation drifted from "all useful human knowledge" to "enough knowledge to do my job".
Both a human and a current large model will bring up unprompted some tax regulation that applies in your district that could matter to your task. A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for, let alone come up with unless specifically prompted about something tax related.
Unless we start having different small models trained for a certain profession for a certain city, I think we'll need those trillion parameters.
What makes it hard to search for? If you tell me you want to open a lemonade stand in East Bumbling Falls, I won't know that city’s municipal ordinances. But I will know that cities tend to have ordinances. So I’d search broadly for what codes that city has. Then I’d go through each one looking for anything that might affect a lemonade stand.
A strong llm without that knowledge can probably spend millions of tokens parsing every single ordinance and give you an answer if you directly ask about that.
But both a local human or an "all knowing" llm when you just tell it about the stand will be like "hmm wasn't there a law about low calorie beverages being tax exempt? Let me check" and after a very precise search come back with a tip that if you sell sugar free lemonade that's tax exempt.
I mean would you get an accountant from another country just because he can use search?