Does this imply we will run out of data to keep up with larger model sizes?
Is there much more data out there than what they’re already using?
Is there much more data out there than what they’re already using?
Consider that a human adolescence is ~9.46x10^6 minutes and a fast speaking rate is ~200words/minute. That sets an upper bound of 1.9 billion words heard during adolescence. ie: human adults are trained on a corpus of less than 1.9B words.
To some extent, more data can offset worse models, but I don't think that's the regieme we're currently in. GPT-3 was trained (on among other languages) 181 billion English words - or about 100 times more words than a human will hear by the time they reach adulthood. How is the human brain able to achieve a higher level of success with 1% of the data?
1. https://github.com/openai/gpt-3/blob/master/dataset_statisti...
Deaf-blind authors would beg to differ.
But yes, a human brain is exposed to lots of other sensory input, and we know from other research that multi-modal models can learn shared representations that benefit from the knowledge of each domain.
In Transformer's favor, at least, they are far closer to tabula rasa than the human brain is and likely have to dedicate a lot of their training time to things that are otherwise "baked" into human brains. For example, humans come pre-packaged with V1 and V2 as part of their visual system, but CNNs and ViTs have to learn those filter packages from scratch.
I agree with you though. Human brains are able to take single instances of experiences and build a wealth of understanding from them in ways that even modern Transformer architectures are not yet able.
All those conversations in the shower were actually regularizers!
The most obvious answer is "the human brain uses a shit-ton more compute", for 18+ years as well.
We spend data, which we have in abundance, to save on compute, which we do not. Even at the most generous low-end estimates of the human brain's computing power, we are only barely there; on the high-end estimates that people in love with the ineffable mysteries of the brain love to cite, we are multiple orders of magnitude away from even the biggest supercomputers matching the brain. So no matter which way you slice it, we are extremely compute-poor.
Feeding a lot of data through an extremely lightweight optimizer like first-order SGDs is one way to cope with lacking compute: https://www.gwern.net/docs/ai/scaling/2013-bottou.pdf Bottou asks why (even in 2013!) is SGD so hard to dethrone when we can empirically see plenty of optimizers like second-order gradient descent algorithms which can beat SGD quite solidly? His observation is that while they are much better than SGD in terms of iterations or _n_, they lose in compute/wallclock because SGD can just go-brrrr through the data much faster than they can.
Some quick googling gives this:
- Generation of an action potential seems to use ~2.5×10^−7 J [0]
- The brain consumes around 20W during normal activity
This seems to imply that there are around 8×10^7, call it 10^8, activations per second [1].
Apparently, the average neuron has 1000 synapses. Let's say each synapse requires 10 mulacc operations per activation. Doing that math gives about 10^12 FLOPs/s [2].
Integrate that over 18 years, and you get roughly 5.7×10^20 FLOPs [3].
PaLM required 2.56×10^24 FLOPs to train [4]. So, we have (way more than) enough compute, we're just not using it efficiently. We're wasting a lot of FLOPs on dense matrix multiplication.
There's plenty of wiggle room in these calculations. I checked over the math, but I'd appreciate if someone would let me know if I've missed something.
[0]: https://link.springer.com/article/10.1007/s11571-018-9503-3
[1]: https://www.wolframalpha.com/input?i2d=true&i=Divide%5B20+W%2C2.5%E2%80%89%C3%97%E2%80%89Power%5B10%2C%E2%88%927%5D+Joules%5D
[2]: https://www.wolframalpha.com/input?i2d=true&i=Power%5B10%2C8%5D+Hz+*+1000+*+10+flop
[3]: https://www.wolframalpha.com/input?i2d=true&i=Power%5B10%2C12%5D+Divide%5BFLOP%2Cs%5D+*+18+years
[4]: https://blog.heim.xyz/palm-training-cost/#:~:text=PaLM%20(2022)-,2.5e24,-10x***- 500M tweets per day
- 30 words/tokens per tweet
- 40% of all tweets thrown away due to being duplicate/spam/bots
= 9B tokens generated per day
It's clear that these models have orders of magnitude too much data already.
It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.
I agree with your main point, but think this analogy isn't an apt one. If you want to see what particles are created at higher energies you kinda need the bigger particle accelerators. (This isn't to say that we shouldn't be investigating lower energy collisions, but at a certain point you do need "bigger colliders" to see new things)
>It's clear that these models have orders of magnitude too much data already.
Seems like a strange claim. The scaling laws are showing that you can still make gains with more data and more parameters.
>It somewhat reminds me of the proposals for larger and larger colliders in the hopes of seeing new physics that is always one collider in the future.
This is literally true though, couldn't find the Higgs without the LHC and most GUT candidates would only start being ruled out at high energy levels.
But then we’ve given up on matching human intelligence which is all about working efficiently with small training data, and certainly training a human does not need anywhere near as much data as GPT-3.
GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective.
If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.
I don't think we should really take so much inspiration from the brain. We didn't make airplanes work by building bird machines so why should we do that here.
>GPT-3 was interesting as a proof-of-concept of what happens when you use a gigantic amount of training data. We don’t need a bigger one until we can figure out how to make a smaller one that is just as effective.
This feels like a non sequitor. We can certainly keep making larger models and we will, because we can continue to make performance gains doing so.
>If scaling laws are telling us to keep putting even more training data into the thing, then the conclusion should be that the architecture is just not working out.
I don't think anyone in the field would agree to this point. Researchers see an easy avenue to gain better performance so they take it. Deepmind's model shows you can get similar results with more refined architecture, but this was released well after GPT-3. When teams significantly advance the state of the art with a much smaller model I think we should take notice but that hasn't happened yet.
It’s not that we should mimic the brain’s implementation, but we should certainly strive to match the brain’s capabilities. One of its outwardly observable capabilities is that it is extremely efficient in the size of the training data set it requires.
Efficiency isn’t an implementation detail, it’s definitional to what “highly intelligent” means.
GPT-3 is not an airplane, it’s a zeppelin. Zeppelins also have scaling laws dictating that a zeppelin should be very very large. Building bigger and bigger zeppelins is one thing, justifying expending resources on gigantic zeppelins by stating the scaling law and concluding that a jet aircraft will magically pop out if you build a big enough zeppelin is quite another.
But generally I think the better analogy is a rocket ship. If we can still go higher and faster with more fuel we should try to do that before we worry about engine efficiency. You have to get to the moon before you can colonize the galaxy.
I have a toy disproof for your claim that this is clear.
Imagine that you are training a ML system using oracle access to Mum. The ML training system can request 10 million representative samples of Mum output, and then we could judge if the ML system has adequately reproduced Mum.
Now also imagine that Mum frequently tells people that Mum knows a 23 letter secret and while mum won't tell people what is outright, she'll answer queries like if a guess is lexographically higher or lower. We could even imagine that the ML has seen Mum's side of some interactions with her doing that.
Would the ML know Mum's secret? No.
Would a child that could interact with Mum? Yes-- after at most ceil(log_alphabet(23)) queries at most, if the child is efficient.
Learning in an interactive context is not the same as learning from written material, so you can't be sure that the fact that children learn english from less text means that a non-interactive ML system could english from the same amount. Q.E.D.
Now, if someone figures out how to efficiently train these natural language models with reinforcement learning...