Gemma 2 27B, one of the top ranked open source models, is ~60GB in size. LLama 405B is about 1TB.
Mind you that they train on likely exabytes of data. That alone should be a strong indication that there is a lot more than memory going on here.
Similarly TBs of Twitter/Reddit/HN add near zero new information per comment.
If anything you can fit an enormous amount of information in 1MB - we just don't need to do it because storage is cheap.
People are claiming that the models sit on a vast archive of every answer to every question. i.e. when you ask it 92384 x 333243 = ?, the model is just pulling from where it has seen that before. Anything else would necessitate some level of reasoning.
Also in my own experience, people are stunned when they learn that the models are not exabytes in size.
The AI pessimist's argument is that there's a huge gap between the compute required for this pattern matching, and the compute required for human level reasoning, so AGI isn't coming anytime soon.
This is exactly what humans do too. Anything more and we need to use tools to externalize state and algorithms. Pen and paper are tools too.
On the other hand general problem solving is, and so far any attempt to replicate it using computer algorithms has more or less failed. So it must be more complex than just some simple heuristics.
Perhaps the answer is just "more compute" but the argument that "because LLMs somewhat resemble human reasoning, we must be really close!" (instead of 25+ years away) seems wishful thinking, when:
(1) LLMs leverage a much bigger knowledge base than any human can memorize, yet
(2) LLMs fail spectacularly at certain problems and behaviours humans find easy
Well, this is what the whole debate is about isn't it? Can LRMs do "general problem solving"? Can humans? What exactly does it mean?
LLMs's huge knowledge base covers for their incapacity to reason under incomplete information, but when you find a gap in their knowledge, they are terrible at recovering from it.