Hacker News Reader

Comment by api | Hacker News Reader

The part about data is the most fascinating, and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data efficient than what we are doing in AI systems.

Anyone who has ever had kids knows this. Little kids learn language and fully developed world models on a tiny amount of data compared to the massive hoards we shovel into our training systems. That data is also noisy and only loosely curated.

The author does mention this but I think he actually doesn’t give it enough importance. A person could read for their entire life and not approach what we shovel into GPT-3.5 or llama2, yet a three year old builds a trained language model from what must be near scratch.

It must be near scratch for the initial condition because the human genome is nowhere near large enough to contain that much “priming.” There can’t be a “base model” analogue or anything like that in there.

View on HN

Kids get full motion video, sound, smell, taste, and touch, not to mention to other less known senses. How that compares to all digitized human knowledge literature, I don't know, but given that we don't have a wide standard for eg smell, I'd hesitate to call their datasets tiny.

To be fair, I don't know how much of that huge input is stored for very long. Human brains are incredible at discarding unimportant information by generalizing information into broad ideas and emotions. Just think back to something that happened a few years ago and try to recall specific details.

Being able to identify and discard unneeded data in realtime seems like a huge step for AI that hasn't been really implemented yet.

Maybe that discarding process and the time it takes to learn it from a combination of so many different senses is the key to developing a human mind to the level of intelligence that it is/has.

Humans are EXTREMELY more energy efficient than computers; an adult human operates at something like 100 Watts on average (much less for children, of course), and that covers not only all of our mental processing, but also all of our physical movement, digestion, tissue regeneration/repair, etc.

At no point in our evolutionary history have we had the leisure to absorb the vast quantities of energy that AI supercomputers can. I think this probably has a lot to do with why we are so effective at learning on little data; even when large quantities of data are available, it seems we may not have the energy to take it on board. We ignore and forget most of it.

Maybe if we made a real effort to develop energy efficient AI, that design limitation would help us develop AI that requires far less training data.

What’s also fascinating is that a child’s brain has additional plasticity/aptitude for learning different things at a different age.

If he/she learns it too late, they’ll never master it. Perfect pitch is an example, ‘perfect’ motor skills for something another. Language another.

Finetuning happens as a teenager/adult.

I actually think that quite the opposite is true.

Yes, children can get some idea of a language, but they need a lot more experience and exposure to reach some level of proficiency in the language and this mostly includes supervised training in the school. They must read multiple books, have conversations etc to be able to really express themselves. It takes 10 years

The difference is that LLM don't try to have such iterative intelligence like humans do but are more like knowledge boxes. Perhaps it would take a lot less data to gain the same level of language understanding the 3 year olds have.

Children are probably better in language-as-communication than current LLMs at relatively young age.

Verbal (and bodily) communication is quite different from written text and its largely artificially imposed structure. Spoken language is very ungrammatical and directly litterated conversational speech is a total mess on writing standards (and a lot of information is lost in litteration).

Even though e.g. Chomskian linguistics make the claim that human language is somehow a special innate syntactic construct (recursive language in the Chomsky hierarchy), I find the empirically stronger view is that the more formal grammars and vocabularies that sometimes are thought of as "language" are very far from how language is actually used when this structure is not imposed.

This quite rigid structure of "correct" written language is also likely a major factor to why LLMs are so good at it.

If a child learned to read at 2 and spent every waking moment reading until 10 it would still be orders of magnitude less than what we use to train even smaller base models (e.g. Mistral-7b).

My 10 year old daughter doesn’t read as much as she should (too much gaming) but gets 100% on science and history tests and definitely would outperform GPT-4 on reasoning. She’d have no problem telling me how many sisters Sally has. :) A human brain consumes 20-40 watts too, not hundreds.

I bet total power consumption for a human brain age 0-10 is substantially less than what it takes to train a medium sized base model. That’d be an interesting bit of math to add.

GPT-4 will win on breadth of knowledge but that’s just because it’s a lossy compression blob containing a dump of the Internet. We are talking about intelligence (GI) here not just being a big dumb know it all.

That theoretical child isn't just reading words though. it's experiencing life in a human body, with a human brain, and getting full force of all their senses and biology of being human. We don't have a good measure for how much data that is, even if the majority of that is being thrown away. This theoretical child reading from 2-10 is reading fewer words, sure, but are they being exposed to far less data? The fonts being used, the smell of the books, how the spine cracks. LLMs don't get that in their datasets, just tokens, which isn't even the letters that make up a word.

> Little kids learn language and fully developed world models on a tiny amount of data compared to the massive hoards we shovel into our training systems.

They don’t learn language in isolation. I doubt a human brain in a jar that was feed only literature would learn very well, if at all.

> and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data efficient than what we are doing in AI systems

Yeah and the brain only needs 20 watts of power.

Reply on news.ycombinator.com