The Scale of the Brain vs. Machine Learning (2022) | Hacker News Reader

The Scale of the Brain vs. Machine Learning (2022)

beren.io

63 pointsreqo36 comments

The part about data is the most fascinating, and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data efficient than what we are doing in AI systems.

Anyone who has ever had kids knows this. Little kids learn language and fully developed world models on a tiny amount of data compared to the massive hoards we shovel into our training systems. That data is also noisy and only loosely curated.

The author does mention this but I think he actually doesn’t give it enough importance. A person could read for their entire life and not approach what we shovel into GPT-3.5 or llama2, yet a three year old builds a trained language model from what must be near scratch.

It must be near scratch for the initial condition because the human genome is nowhere near large enough to contain that much “priming.” There can’t be a “base model” analogue or anything like that in there.

> and I think it’s rather obvious that whatever the brain is doing with learning must be quite a bit more data efficient than what we are doing in AI systems

Yeah and the brain only needs 20 watts of power.

What’s also fascinating is that a child’s brain has additional plasticity/aptitude for learning different things at a different age.

If he/she learns it too late, they’ll never master it. Perfect pitch is an example, ‘perfect’ motor skills for something another. Language another.

Finetuning happens as a teenager/adult.

> Little kids learn language and fully developed world models on a tiny amount of data compared to the massive hoards we shovel into our training systems.

They don’t learn language in isolation. I doubt a human brain in a jar that was feed only literature would learn very well, if at all.

Kids get full motion video, sound, smell, taste, and touch, not to mention to other less known senses. How that compares to all digitized human knowledge literature, I don't know, but given that we don't have a wide standard for eg smell, I'd hesitate to call their datasets tiny.

To be fair, I don't know how much of that huge input is stored for very long. Human brains are incredible at discarding unimportant information by generalizing information into broad ideas and emotions. Just think back to something that happened a few years ago and try to recall specific details.

Being able to identify and discard unneeded data in realtime seems like a huge step for AI that hasn't been really implemented yet.

Maybe that discarding process and the time it takes to learn it from a combination of so many different senses is the key to developing a human mind to the level of intelligence that it is/has.

Humans are EXTREMELY more energy efficient than computers; an adult human operates at something like 100 Watts on average (much less for children, of course), and that covers not only all of our mental processing, but also all of our physical movement, digestion, tissue regeneration/repair, etc.

At no point in our evolutionary history have we had the leisure to absorb the vast quantities of energy that AI supercomputers can. I think this probably has a lot to do with why we are so effective at learning on little data; even when large quantities of data are available, it seems we may not have the energy to take it on board. We ignore and forget most of it.

Maybe if we made a real effort to develop energy efficient AI, that design limitation would help us develop AI that requires far less training data.

I actually think that quite the opposite is true.

Yes, children can get some idea of a language, but they need a lot more experience and exposure to reach some level of proficiency in the language and this mostly includes supervised training in the school. They must read multiple books, have conversations etc to be able to really express themselves. It takes 10 years

The difference is that LLM don't try to have such iterative intelligence like humans do but are more like knowledge boxes. Perhaps it would take a lot less data to gain the same level of language understanding the 3 year olds have.

Children are probably better in language-as-communication than current LLMs at relatively young age.

Verbal (and bodily) communication is quite different from written text and its largely artificially imposed structure. Spoken language is very ungrammatical and directly litterated conversational speech is a total mess on writing standards (and a lot of information is lost in litteration).

Even though e.g. Chomskian linguistics make the claim that human language is somehow a special innate syntactic construct (recursive language in the Chomsky hierarchy), I find the empirically stronger view is that the more formal grammars and vocabularies that sometimes are thought of as "language" are very far from how language is actually used when this structure is not imposed.

This quite rigid structure of "correct" written language is also likely a major factor to why LLMs are so good at it.

If a child learned to read at 2 and spent every waking moment reading until 10 it would still be orders of magnitude less than what we use to train even smaller base models (e.g. Mistral-7b).

My 10 year old daughter doesn’t read as much as she should (too much gaming) but gets 100% on science and history tests and definitely would outperform GPT-4 on reasoning. She’d have no problem telling me how many sisters Sally has. :) A human brain consumes 20-40 watts too, not hundreds.

I bet total power consumption for a human brain age 0-10 is substantially less than what it takes to train a medium sized base model. That’d be an interesting bit of math to add.

GPT-4 will win on breadth of knowledge but that’s just because it’s a lossy compression blob containing a dump of the Internet. We are talking about intelligence (GI) here not just being a big dumb know it all.

That theoretical child isn't just reading words though. it's experiencing life in a human body, with a human brain, and getting full force of all their senses and biology of being human. We don't have a good measure for how much data that is, even if the majority of that is being thrown away. This theoretical child reading from 2-10 is reading fewer words, sure, but are they being exposed to far less data? The fonts being used, the smell of the books, how the spine cracks. LLMs don't get that in their datasets, just tokens, which isn't even the letters that make up a word.

These exercises are fun and may shed some light on comparable complexity of machines vs animals, but they make the (unrealistic) assumption that the computational architectures are comparable.

Brains (and the nervous system and body in general) have many crucial differences from the (current mainstream) ANNs, beyond the Assumptions listed in TFA. Brain connections are higly recurrent at multiple levels, they are magnitudes slower than GPUs but they are dynamical in the state (e.g. temporal summation), there's a lot of information encoded outside the synaptic connections (e.g. concentration of transmitters/modulators in intercellular fluid), the synaptic connections are more complicated than just a scalar weight (modulated by transmitters etc), etc etc.

Also all the rage in contemporary views of animal intelligence is "embodiment" that stresses that there's more to animal intelligence than the brain.

Similar, quite apples to oranges, comparisons were made to von Neuman computation architectures (CPU FLOPS and hard drive/memory bits) when the "computer metaphor" of the brain/mind was the trend. In general there's a tendency to compare human intelligence to whatever is seen as the state-of-the-art in machines (e.g. clockworks and telephone networks back in their respective days). These probably give some intuition, but shouldn't be taken too seriously for e.g. projecting when machines and humans are at parity.

Human intelligence is really a global phenomenon, not just an individual phenomenon. A lot of people outsource intelligence and specialize in labor or beauty. even intelligence requires so much specialization. Humans are a mixture of experts.

A physical laborer or pretty person who relies on those traits could generally switch and learn to code or run a business if pressured (either by desire or need) to do so.

We specialize but we all have similar brains.

The last point about IQ at the very end of the article is a thought I’ve had before and is a reason I remain skeptical of the importance assigned to IQ by many. IQ seems to be a real measure of something but the variance both within and between people and the way that variance behaves is sus. I think there’s something more complex going on with IQ, probably more than one thing.

My theory having studied this for a long time is that higher IQ simply means that you have the ability to forecast further into the future with higher accuracy on a variety of things.

Conceptualized differently the “simulator“ in your brain that hallucinates futures is more accurate for smarter people than it is for let’s say less capable people. And some people don’t seem to have a simulator at all.

The more things and further into the future you can forecast the “smarter” you are.

I find this compelling because exceptionally smart people also seem to have exceptionally different ways of interpreting the future and also our more likely to hallucinate stuff that is totally false or incoherent. This is why you often see “geniuses” in the physical sciences struggle with social things.

The same as true people who are geniuses, socially often struggle, with other concepts outside of their narrow expertise, but are generally better and faster at learning or at least pretending better than other people with lower IQ at those general things.

Don't geniuses also tend to score better on stuff like reaction times as well though, rout calculation, etc though?

Yeah and are generally more creative overall enjoying mastering another fine art of some sort different than their line of work

A lot of humanity hasn't achieved intelligence very well. Chance and neurodivergence causes enough intelligence for the rest of humanity to reap the benefits, but a lot of people are sort of dumb.

It's not necessarily a bad thing, and those people still deserve to live happy lives, of course!

I think you’re defining intelligence too strictly, and in relation to work (probably). Consider how much shared intelligence is present in your Culture, and how complicated the social rituals inevitably become. Humans are inherently damn smart, IMO

> in relation to work

I can't work very well at all, but I still notice that many people just live by "ignorance is bliss" to the point where they almost try not to know anything. They don't want to know why anything works the way it does, they don't ever reflect on themselves or their actions, they don't ever perform much introspection or anything. It's almost like they operate purely on instinct. Is there a name for this phenomenon?

Bad epistemic practices/preferences/strategies. Or just “burnt out” lol

IME, it seems to be sort of a pathological thing and not just burnout? When you burn out, you still know everything you've learned, but some people just are not self aware, do not care about it, and do not want to know how things work or why. The sort of people who will throw a fit if a software update moves a button, because they semingly don't have the knowledge or intuition to spend 10 seconds finding where it went. This baffles me, because I know plenty of neurotypical people who can do this, so it's not like it's a neurodivergence thing, it's just that some people seemingly are not curious about the world and do not even understand how to seek knowledge.

Maybe this is an education issue? Back when I was in school, they never taught me how to learn, they just wanted me to memorize things and then repeat them back. I don't know how that would explain the difference though, since plenty of neurotypicals are served "well enough" by school. Comparing my own experiences is sort of a slippery slope since I'm neurodivergent myself.

Not an expert by any means but aren’t humans pretty much all similarly intelligent. There are geniuses of course but almost everyone knows the hard stuff of surviving, speaking, interacting, using tools etc. The logical thinky stuff that operates on top of the mind is pretty shallow part of the whole.

Like Usain Bolt runs a lot faster than me, but hey we both run so I would argue I am 95% of his performance level even though it takes twice as long for me to go 100m. I have not outsourced running to him.

This is a good insight. People tend to compare capabilities to other people, but these differences are really minor when comparing e.g. cross species or to machines.

E.g. Magnus Carlsen will beat me every time in chess, but compared to Stockfish me and Magnus are roughly on the same level.

The author assumes that evolution should somehow optimize data bandwidth but this is a super faulty understanding of how evolution works.

A trait like intelligence is only valuable to the extent it increases an organisms fitness or number of nonsterile offspring. Witness millions of years animal evolution of organisms that had better or equivalent visual systems to humans but paltry brains.

There are some possible examples of data augmentation that might be interesting to consider though. For example, our eyes are constantly making small involuntary movements at a rate we mostly do not consciously notice. In that sense we're getting a feed of slightly transformed and translated images every few milliseconds

This is a fascinating article to me. I love it when a less "techy" but still very much nerdy blog post makes it here. It allows me to peer into things I wouldn't know how to find otherwise.

Thank you!

This always feels like an apple to orange comparison that people who are clearly not trained in neuroscience probably shouldn't be making. That we call them artificial neural networks is more a coincidence than a suggestion that these things work like the brain at all. He mentions a lot of this in his section about assumptions but seems to gloss over how all those points in aggregate make these kinds of comparisons worthless.

I agree that the comparison is apple to orange (or apple to pine tree). I'm not a neuroscientist, but it's a hobby area that I read a lot of the research about the brain and the cells in the brain, etc. With each new study it seems that the iceberg just keeps getting bigger and neuroscientists still just understand a minority.

Some interesting examples: 1-Individual Purkinje cells, in isolation, learn to respond to input patterns. Meaning no synapses or other cells, the learning is happening within that cell.

2-Until relatively recently, the equipment to detect electrical activity wasn't sensitive enough so it wasn't known that dendrites have localized spiking. The collection of inputs to the dendrites are getting pre-processed either linearly or non-linearly (depending on region/function/type of cell/etc.) before being forwarded to the neurons nucleus.

3-Astrocytes have intra and inter calcium wave signaling. The intra cell signaling is localized within "proceses" (small extensions and/or compartments of the larger cell). This signaling has now been linked to sensory information processing as well as learning, meaning it's computational and separate from neuronal computation. The equipment to directly capture this activity doesn't exist so understanding is limited (studies have been based on knocking out certain capabilities and seeing the result vs direct sampling of the waves).

My opinion on comparisons: 1-A single cell in the brain is performing many more valuable computations than a single artificial neuron.

2-The brain probably also has many excess cells so a loss of a single cell doesn't have an impact. Which partially negates #1.

3-The way the brain process information could be more effective than an ANN for some functions, and it could be less effective for other functions. I think there are too many unknowns in this area to try to line it all up and compare.

> However, current visual recognition and image generation is already really good, and seems closer to parity with humans

I doubt that visual recognition is close to parity with humans. I think someone would have to define precisely what is included in "visual recognition" and what is not. It seems that "recognition" for humans means more (resolves more info about the object/context/environment) than what it means to an ANN.

The author admits in the introduction that this is an attempt at “napkin math.” It’s an attempt to get a sense of whether current systems might even be in the ballpark.

> The brain is currently our sole example of an AGI.

I'm sorry. What? If you're starting like this I will stop reading.

Why?

AGI = Artificial General Intelligence.

By definition we are not AGI. Maybe GI, but when the author has trouble articulating basic things they lose credibility and I lose interest.

With that pedantic an approach you are missing out on an otherwise fairly informative piece.

I think words matter and it's important that we all agree on the meaning if we want to convey ideas. My approach may be pedantic - but it also saves me from wasting my time reading an article that my be full of idiosyncrasies. It's okay if other people take a stab at parsing the article and/or derive value from it through.

I think the author may be taking the (unspoken) position that we are just a physical system, no cartesian dualism, so comparisons of scale (accepting that processing mechanisms are vastly different) may be relevant. So Broca’s and Wernicke’s may be the scale of current LLMs, but that leaves other mechanisms in our GI, like the rest of the brain + processing elsewhere eg: the gut - and the social dimension of intelligence, on the table. I still found it an interesting read - just for the scale discussion.

Reply on news.ycombinator.com