Hacker News Reader

Comment by jonplackett | Hacker News Reader

Yeah I'm not totally convinced humans don't have a tremendous amount of training data - interacting with the world for years with constant input from all our senses and parental corrections. I bet if you add up that data it's a lot.

But once we are partially trained, training more requires a lot less.

View on HN

> I bet if you add up that data it's a lot.

Let's Fermi estimate that.

A 4k video stream is about 50 megabits/second. Let's say that humans have the equivalent of two of those going during waking hours, one for vision and one for everything else. Humans are awake for 18 hours/day, and we'll say a human's training is 'complete' at 25.

Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.

There's plenty of reason to think that we don't learn effectively from all of this data (there's lots of redundancy, for example), but at the grossest orders of magnitude you seem to be right that at least in theory we have access to a tremendous amount of data.

> Let's Fermi estimate that.

I couldn't agree more. Fermi estimates are underused.

> Multiply that together, and you end up with 1.8e17 bytes, or 180 petabytes of data.

And for comparison, GPT-4 is estimated by Bill Dally (NVIDIA) at around 10^12 or 10^13 tokens [1]. Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 50 terabyte [Edited after comment below].

As a side note, I would guess that GPT-4 knows more "things" if you would be able to count them all up. For example, it knows more languages, more facts about cities, more facts about persons. However, people know way more inside their specialization.

[1]: https://youtu.be/kLiwvnr4L80?si=77pXmIBmlp8dsSCG&t=349

> Let's assume about 1 word per token and 5 characters per word. Furthermore, US-ASCII tokens requires one byte per character in UTF-8. So, that gives about 40 petabyte.

Thanks, I was looking for that estimate but could not quickly find it. (Although are you sure about that 'petabyte' figure? I see 1e13 * 5 = 50e13 = 50 TB?)

I love these sorts of bulk comparisons, since they let us start to reason about how much humans learn from "each bit of experience" versus LLMs. On one hand humans process a lot of sensory information (most of it by discarding, mind), but on the other humans do a lot of 'active learning' whereby experience is directly related to actions. LLMs don't have the latter, since they're trained on passive feeds of tokens.

Your estimate is using compressed video rates, which might be a argued way to look at the way that the brain processes and fills in presumably "already known" or inferred information. I don't know enough about this subject to make that argument.

Uncompressed rates are easy to calculate if wanting to adjust your approximation:

24-bit, 4K UHD @ 60 fps: 24 × 3840 × 2160 × 60 = 11.9 Gbit/s.

Children are awake 12 hours per day, a 90 minute movie used to fit on a CD-ROM, 8 700MB CD-ROMs times 365 is 2TB per year. Alexander the Great became king aged 20, he seemed trained enough by then, so that's 400TB, 4e14 bytes of data.

Pretty unconvinced by this argument, it mostly hinges on the video stream size.

I think it is an estimate. Yours is on the extreme low end of things. The GP was using 4K as a standin for touch, smell, taste, pain, hunger, pleasure, etc. Those are a bit more complex than just videos and you can't encode those on a CD.

Regardless, yours still came out to be an order of magnitude higher than what GPT was trained on. So I guess the original argument makes some sense.

So much of that data is totally useless though. Most of the data we get is visual and I would argue that visual data is some of the least useful data (with respect to volume). Think about the amount of time we're looking at blank colours with nothing to learn from. Once you've seen one wall, one chair, one table etc, there's not much left to know about those things. An encyclopedia though for example is much less data than a few hours of high res video of clouds passing by yet clearly its orders of magnitude information rich.

Also have to disagree here. You see that one object yes, but you see it from 10000000s of different angles and distances. In different settings. In different lighting conditions. With different arrangements. And you see the commonalities and differences. You poke it. prod it. Hit it. Break it. You listen to it.

This is the basis for 'common sense' and I’m pretty sure everything else needs that as the foundation.

Go watch a child learning and you'll see a hell of a lot of this going on. They want to play with the same thing over and over and over again. They want to watch the same movie over and over and over again. Or the same song over and over and over and over and over again.

This just sounds so hilariously wrong start to finish. That any of the data you collect is useless is already a big [citation needed]

This is completely and fundamentally an incorrect approach from start to finish. The human body - and the human mind - do have electrical and logic components but they are absolutely not digital logic. We do not see in “pixels”. The human mind is an analog process. Analog computing is insanely powerful and exponentially more efficient (time, energy, bandwidth) than digital computing but it is ridiculously hard to pack or compose, difficult to build with, limited in the ability to perform imperative logic with, etc.

You cannot compare what the human eyes/brain/body does with analog facilities to its digital “equivalent” and then “math it” from there.

Also why trying to replicate the human brain with digital logic (current AI approach) is so insanely expensive.

You missed the point of the exercise. Of course it's extremely difficult to compare the two, but the question was: do humans get nearly as much training data as LLMs do? This analysis is good enough to say "actually humans receive much more raw input data than LLMs during their 'training period'." You're concerned with what the brain is doing with all that data compared to LLMs, but that's not the point of the exercise.

No, you can’t even compare that because the information isn’t packetized. Again, we don’t see in pixels so you can’t just consider sensory input in that manner.

What about using energy cost as a stand-in for the data?

The net energy cost of training and maintaining humans vs. training and maintaining (updating and augmenting) AI models.

The human body is incredibly efficient but there’s always a larger ramp-up cost.

Nit: The bandwidth of your optic nerve is only about 10 kilobits per second. You think you're seeing in 4K but most of it is synthetic.

Academic sources I’ve seen are in the 10Mbps range, e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1564115/ (estimating 875kbps for a guinea pig and ~10x that for a human).

That cannot be right.

10 kilobits per second would only get you some awful telephone quality audio. Even super low res video would be orders of magnitude more than that.

Someone added it up:

https://osf.io/preprints/psyarxiv/qzbgx

> Large language models show intriguing emergent behaviors, yet they receive around 4-5 orders of magnitude more language data than human children.

_language_ data.

Humans get a lot more input that just language is the point. We get to see language out in the physical world. They have the (really quite difficult) task of inferring all the the real world and its connections, common sense, laws of physics, while locked in a box reading wikipedia.

How capable do you think a child would be if you kept them in a dark box and just spoke to them through the keyhole? (not recommending this experiment)

Reply on news.ycombinator.com