This is assuming we are learning nothing during sleep, which probably isn't true.
By the time a person is 21 years old, they have been trained on at least 1 petabyte of data.
By two years old, about 125TB of data. It makes LLMs look quite good in comparison.
Call 10^10 \approx 2^40 for convenience, and 8000 \approx 2^13, which gives us a 2^53 entropy estimate, or about a petabyte of information as an estimate of what the human brain can store (discounting more exotic theories of memory stored in DNA or some such).
[0] https://en.wikipedia.org/wiki/Human_brain#Microanatomy
[1] https://psychology.stackexchange.com/questions/7967/how-many...
A single pyramid neuron in the neocortex might be more comparable to a multilayer neural net.
https://www.biorxiv.org/content/10.1101/2021.10.25.465651v1....
We don't understand how they work at the subatomic level simply because human understanding of the subatomic world is not complete, but even just at the atomic level a single neuron is massively more complex than anything humans have created.
Going up to the molecular level, even that is staggeringly more complex than the incredibly simple abstractions that make up a neural net.
Is what happens in the brain at the molecular, atomic, or subatomic levels relevant or necessary to intelligence and consciousness? We just don't know yet, but we do know all of that is far more complex and very different from the simple abstractions that are used for neural nets and LLMs.
The back of a napkin calculations in this thread don't even begin to do justice to the tremendous amount of "calculation" or "storage" that happens in the human brain.
In contrast the data we collect through our nervous system is rich and meaningful and far deeper than just raw text. We can even manipulate the environment as we learn to facilitate faster learning eg. pick up a ball and throw it, rather than just watch videos of balls being thrown.
Our object recognition is trained pretty quickly. And we sleep (certainly as a child) more than 6 hours per day. But we don't learn much from just looking at pictures.
> 18 hours of reinforcement learning
You made that up.
> It makes LLMs look quite good in comparison.
So, exposing an LLM to a lot of video will make them understand language?
Even sound alone (uncompressed CD quality stereo) is 3TB/year.
Some deaf/blind people can read braille — they learn fine too.
Much less data than you might think.
Bare in mind most people can run the brain on 2500kCal/day.
And so on.
Which is, in turn, 2900 W·h, or 2.5 times less than one A100 card working round the clock (300W·24h)
The significant thing is the reinforcement. Without it you have to resort to the openai brainlet style 10tb of text training.
If their model had reinforcement built in then you could just plop it in front of a person or on reddit and it would rapidly self-learn on a fraction of the data. Their training model relies on it learning purely based on observations rather than interaction which is inefficient.
(That said, DL architectures are obviously wildly different from how the human brain works. E.g. backprop is physically impossible.)
I'm sure there's a lot more to it than this, but maybe one factor that makes humans a lot more data efficient is the multimodal input we receive.
If that's the case, imagine how much better things could get when we train with all the videos, podcasts, radio etc in the world, in addition to all the text out there!
And it's been optimized over billions of years.
You can look at it the other way around - dopamine and other neuro transmitters as poor approximation of backpropagation. It has many flaws for example tight harmful loops ie. addictions.
Majority of brain work is ignoring irrelevant information (attention) and small scale hallucinations (we don't see world as is but slightly hallucinated to keep it stable - ie. they way brain processes blinking <<turns off>>, you can peek at those nuances with ie. optical illusions etc).
One of missing bits in neural nets may be reusing its output as input (embedded in inference itself, not poor mans re-prompting).
Once it's sorted out I'd argue the performance will skyrocket and give opportunity to massive optimisations ie. embedding things like known functions - imagine brain which has known, very narrowed, available functions at its disposal - all mathematical functions on numbers, logic, optimal sorting etc. Imagine if as thinking human you'd have access to accurate functions - the sky is a limit.
Bayes formula as a built in primitive. You don't need to know much of statistics to see how limited humans are at processing information because estimating posterior updates is so expensive for them.
Thinking in terms of raw probabilities would be very alien to most humans, but could easily be technically superior for making plans.
This is only true when considering single performance axis like pixel resolution. When you consider the corpus of power efficiency, jitter resolution enhancement, dynamic contrast, performance per volume, etc. We aren’t close to building something as capable.
It's pretty clear that humans, unlike LLMs, use external sensory data (the only external data we have, when you cut through it all) when they produce speech, as evidenced by the fact that they don't speak falsehoods that don't mesh with their internal data model. LLMs have such a weak model of reality -- it's whatever “sounds right” -- that they speak falsehoods all the time. The only way to give an LLM sensory data would be to encode every sensory experience people have into text.
I don’t have access to any other humans internal data model, but the indirect evidence I do have suggests that they do, in fact, speak falsehoods that don’t mesh with their internal data model for a variety of strategic purposes.
A chimpanzee can listen to humans as much as it wants, and it will still not pick up much of the language.
It's deeper than that. Our brains are optimized for a lot general human functions. Learning language is one of them. There's a whole section of the brain dedicated to it and other things.
There's also a lot of vital biological information encoded in DNA/RNA. We are not even close to starting from scratch.
What weights are you referring to ?
Obviously newborns need to develop and take in stimulus before they "know" anything, so the initial conditions are not sufficient, but they are obviously necessary to make the limited learning useful.
As far my understanding goes to very large degree it is unknown how to model such dynamics, where it would be possible to start with no spiking neurons and evolve effective/stable learning behavior from small number of examples/experiences.
There may be many roads to intelligence. The path biology and evolution took may just be one such path.
Are they coded in DNA, or do epigenetic factors, the environment cells grow in, adjacent cells, their interaction, etc, play a role here as well?
Does this refer to DNA -> MRNA -> to protein or is there some other mechanism here?
Also, since you asked, I would like to mention protein interactions with cellular components and tissues. These can be argued as just proteins interacting with each other but the extreme complexity of these higher levels and the unique phenotypes arising from them make me think they are deserved to be treated as another layer of "data decompression".
Yes, we use now magnitudes more oil than we used to 120 years ago. But back then oil was used for cooking and lamps, while now it is used for so many more things. Same goes for data. It is the new oil :).
But regardless, the architecture of the human brain doesn't have to be the only way to get to AGI (not that LLMs are necessarily the way)
So does Wikipedia. But that's not a good model of the human brain either.
> I don't think it's quite an apples to apples comparison.
Yes. That is exactly my point. Despite the superficially similar I/O behavior, the two systems are very different under the hood.
There's some reasons to suspect this is at least partially true, but to what extent is unknown and contraversial.
It doesn't say anything about that knowledge being precoded.
The fact that human brains are capable of learning things that animal brains cannot, is a clear indicator that it is already "trained" to a significant degree.
Human "learning" seems to be much more analogous to "fine tuning and memory storage/retrieval", than actual training.
Why can't a macaque teach a calculus class? Why can't its "empty" brain be taught to do something like that?
These LLMs are also trained on arbitrary books. When acquiring new languages, we use educational materials that are specifically created to facilitate an understanding of language. Not just arbitrary books in random order.
I don't care if my model needs an exabyte of RAM if I can just go and buy that much RAM one day.
For the record, I'm not saying LLMs are not a huge step forward. They are. But they are not -- and cannot be -- the whole answer.
That helps tremendously compared to a tabula rasa.
If you mean a lot of accuracy, that's obvious and doesn't really need argument. And this new fact doesn't change the argument.
If you mean a more moderate amount of accuracy, this isn't proof either way. Human brains take in less text but they put a lot more processing into it.