Mammals dream about the world they are about to experience before they are born
news.yale.edu
news.yale.edu
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
Are these innate patterns of activity priming mammalian brains in a similar way?
A newborn foal can stand within 55 minutes of being born and can walk or run within 90 minutes. That is crazy fast training speed. Is it training initialization or is it a form of transfer learning?
Most human babies take 9-18 months to walk, and this is commonly attributed to the fact that human babies have immature brains that cannot mature due to birth canal constraints, which is constrained by pelvis size which is constrained by the need to walk upright.
I have wondered about how true this is. Maybe our brains are just so much more powerful, or capable of much deeper understanding, that we require a much lower initial learning rate.
The lottery ticket could explain why some children learn to walk at 6 months, while others are closer to 24 months, with no different in intelligence or motor skills in later life.
Edit: As some people below have correctly pointed out, human babies could not walk within an hour for physiological reasons, but nor do they exhibit the basic motor skills, spatial reasoning or image processing required for walking. Many human babies even struggle to feed for the first 24-48 hours or longer.
Meanwhile baboon babies can hold onto their mother from birth while the mother climbs trees.
I strongly doubt that human babies are physiologically capable of walking before 6-9 months regardless of their level of brain development. The entire baby is born many months "early" compared with other mammals due to the limitations of the human pelvis.
[1] https://www.scarymommy.com/complete-shock-early-walking-baby...
Our eldest wasn't interested in crawling really and went almost straight to walking but not until about 14 months IIRC (it was > 10 years back).
We did baby sign (a sort of simplified BSL; would recommend) and some people suggested it would retard speech - same child modified a sign before 12 months and so could tell us he needed a poo/wee and we'd sit him on the potty to do it all before he was able to talk - saved me changing a lot of stinky nappies! Longitudinally, our kids used baby sign and did/are doing well at school and have all been ahead with language skills (but that's not accounting for confounding variables).
It's curious to see infant developmental stages becoming more and more to be compared to AI/robot development.
Have you ever seen a new-born baby's legs?
They aren't even remotely strong enough to walk, no matter what the brain tells them. It takes those months to develop leg muscles. New-borns don't have them.
It's absolutely not restricted by the brain size.
Anyway, hard to conclude, just think your logic is missing a little.
The brain size needs a huge head, so in a sense it is.
All the factors are interwined. The growth of the brain seems to have been a very fast evolution, The pelvis had no time to adapt, so giving birth early was the final solution.
With that in mind, I don't find fair to shame the pelvis, or the brain. The weakest link is the legs, but it's not their fault either since they had no time to mature.
Give one of those cocky foals a head of proportional weight and allow them to use only two limbs and see who's stumbling now!
Edit: newborns can't even balance their heads on their necks, they're extremely fragile.
The balance comes much later than the strength.
If you start occasionally holding your baby up by the thighs when they are 4–5 months old (their back/abdomen muscles at that point are typically plenty strong), after about a month they will have the balance to hold their torso upright when sitting on an adult’s shoulders, for at least a while. By age 8–9 months they will be very stable and good at balancing for long stretches of time. (I recommend any new parent try this, not only for the baby’s sake but because carrying a baby on shoulders is dramatically less tiring for an adult than carrying a baby in arms, and less hassle than a stroller or baby carrier.)
People generally hold their babies from the top (e.g. with hands in armpits), in a way that they are passively stable. If you want to train someone’s balance, you want to force them to actively stabilize themselves from a default-unstable position at least a bit every day. Babies have very short torsos and small heads compared to older kids, with the effect that their torsos have a much smaller moment of inertia about the waist: it takes several times more muscle strength for a 3-year-old to lift their torso up.
If you get a few 3-year-olds and e.g. get one to practice kicking a ball for 10 minutes per day, another to practice riding a 2-wheeled scooter, another to practice hanging from monkey bars, and another to practice swimming, after a few months there will be a wide gulf between their abilities at those skills.
Do you mean baby sits on my shoulder with his legs around my neck, I am holding on to his thighs, and he can hold on to my head with arms? We both face forward in same direction.
Then after a month or two of practice (a few minutes per day, nothing too serious here) you put your baby on your shoulders, holding his ankles, and he will be able to hold his torso up with his abdominal muscles for at least a few minutes at a time. Practice this for a few more weeks and you will be able to walk around town with the baby on your shoulders.
Is it? If we ignore compute requirements, how long would a good algorithm take to learn something like balancing a quadrupedal robot, based on real-time feedback to its outputs? It is a simpler problem, but drone flight software based on learning, can re-learn how to fly a quadcopter after something like losing a propeller and a sudden shift in weight distribution, in a few seconds.
A robot could be built that learns how to adapt to something like a limb being lost or added or half its weight shifted to the other end. It could probably learn an approximate optimal to move under those conditions in just seconds, as well. Though, I suspect that foal might actually be nearly as adaptable. Adult horses, or humans... not so much. We have many overlapping models of how to move, probably. I would take only seconds to adapt to half my weight being added on my shoulders, in the same ballpark as our best robotic systems. But if my left leg grew four inches it would take me a lot longer than the robot to learn and internalize the best way to move again. I could barely keep up with that when it was just a few inches a year when I was a kid.
Seems easy but why don't we have self driving cars by now?
Walking it not just a mechanical task. Your quadcopter is not learning to interpret raw electrical signals from an IMS and raw electrical signals from a CMOS sensor. The training is also probably done in a simulator, where if it breaks, it is reset and training continues.
A horse is learning to interpret inner ear signals and signals from the optical nerve AND the using those to build a spatial map of the environment. Only then it is actually trying to walk.
A 2 hour old baby horse can not only run on not-flat terrain but is spatially and environmentally aware. It has taken years for Boston Dynamics to develop a platform that is equivalent to a pack horse and even then probably can't respond independently to never-seen-before threats or challenges.
Trying to build a neural net or set of neural nets that can do all of those jobs is a daunting task and no one has the time to train the artificial neural network for 15 years. that's why we don't have safe self driving cares yet.
as for the horse, well 50 millions of years of evolution on constantly refined genetic algorithm is hard to beat and that was on top of the evolution of the eohippus as a starting point
Firstly, the drone software is not re-learning anything; when a baby walks unaided for the first time and you place a rattle in their hands, do you really refer to that as "re-learning walking"?
Secondly, forget about computational requirements and just look at how many iterations have to be run to reach success; mammals (and humans specifically) can "learn" to recognise an entire class of something just from one or two images of that something. Learning something else doesn't "forget" the old thing. A NN trained to recognise faces in images can't, to me knowledge, be trained to also balance a bi-pedal robot, or ride a bicycle.
IOW, what I see currently is that NNs are very one-dimensional - the net that becomes optimised for one task becomes unoptimised for that task if you subsequently add new tasks.
The way I understand it is that is not so much required as not selected for. We could certainly be able to walk at one day old if evolutionary pressure made it necessary, but it prioritized other aspects in our cognitive development to survive so we don't. It's not as if we say "oh, horses can walk early and therefore are incapable of rational thought" we evaluate them on their own merits.
Probably neither. How does the human baby "learn" to breath almost immediately after birth? I suspect the answer here is also the same.
I doubt it's randomly initialized. The brain has centers for motion which might be honed in at birth for these animals.
Let's not ignore genetics, which pre-populate weights and subnetworks in the form of motion-control centers in the brain. Once you consider the gene-brain interaction over the course of evolution as part of your training, the huge data required for modern ML starts to make more sense.
https://www.stanfordchildrens.org/en/topic/default?id=newbor...
Perhaps the most relevant to this discussion is the "stepping" reflex. It isn't really stepping but if placed in a standing-like position babies will often raise one foot, put it down, then the other. The most interesting part to me is that this reflex stops by two months. Makes me wonder if it's an evolutionary leftover from our days as monkeys or something like that. It's all so fascinating.
I read that both failure to demonstrate these reflexes while newborn, as well as continued existence past a few months old, are signs that something might not be entirely right with the baby.
Interestingly, the EGG hypothesis postulates that the constraining factor is not pelvis size, but rather that neonates consume so much energy that we give birth just before a women's metabolic limits would otherwise be overrun (leading to both maternal and infant death). Based on this, the increased energy demands of our larger brains mean that we have to sacrifice neonatal motor skills or drastically improve the metabolic efficiency of adult women.
https://www.pnas.org/content/109/38/15212
> In humans, maximum sustained metabolic rate is thought to be 2.0–2.5× basal metabolic rate (BMR).
> By 9 mo, metabolic demands of the fetus threaten to push maternal energy requirements beyond 2.1× BMR. Extending gestation by even one month would likely require metabolic investment beyond the mother’s capacity. Instead, the mother delivers and the neonate’s growth rate slows relative to its fetal growth rate, keeping both the offspring’s and the mother’s energy requirements in check.
Worth noting how massive the growth rate is I think: Even after birth, the baby gains weight faster than it ever will again in its life. In absolute terms! In relative terms it's even more insane.
A 12-year-old boy going through puberty will go from 90lb to 100lb over a year, an increase of 10lb or ~10%.
The average baby boy will be born at 7.5lb and weigh 20lb at one year old, an increase of 12.5lb or ~270%. Nearly triple the size!
The growth might seem impressive when you do percentages of a very small number, but it's still a very small number overall.
If not--I am very lightweight right now, and I'm trying to put on body mass by eating more and exercising. While my volume will increase, my main growth is in density. My growth is not in fat, but actual muscle mass. Is this growth? If so, I have gone from 125lb to 140lb in ~2 months, which in absolute terms is obviously more than the baby (my goal is 160lb by the end of the year).
A few years ago, I went from ~130lb to 165 lb in three months--blame the free food at FAANG cafes during an internship. But this might be closer to what you were saying earlier--putting on fat.
Edit. I guess in the original statement saying that a person will never gain weight at this rate again there should be the explicit assumption that we're talking about natural growth, not growth due to external factors (ie more eating--intentional or not). In that case, that statement would be true.
But in that sense, of course a baby will have more growth than an adult--isn't it reverse exponential?
Average height at age 12: 59". At age 13: 62". 3" of growth.
At birth: 19". At one year: 29". 11" of growth.
Try matching that just by eating more.
(obviously, yes, it's easier for a small thing to grow than a large thing. But it's still crazy fast. Imagine being 50% taller in a year)
This seems weird. Fully breastfed babies require even more calories from the mother. (They are bigger than unborn babies, and they lose more heat.)
Growth rate slows after birth, so that would account for some of that. Mother’s also would not need to carry around the weight of the baby, and would be more able to provide for the child.
Though it's not 24/7.
In contrast, by the third trimester of pregnancy, your total blood volume has increased by approximately 50% (3-4 pounds of additional blood), not to mention an extra 2-3 pounds of retained fluid, 2 pounds of amniotic fluid, and 1.5 pounds of placenta. All told, this is another ~10 pounds on top of the baby. Additionally, the mother / placenta is performing all respiratory and blood filtration functions on behalf of the fetus (the kidneys don't really do much prior to birth other than produce urine).
> https://www.mayoclinic.org/healthy-lifestyle/pregnancy-week-...
Personally, while breastfeeding and newborn care was exhausting, it felt more akin to training for an athletic event vs having another entity symbiotically tied to my system like pregnancy.
I was mostly just remarking on the energy-demand hypothesis advanced in the comment above.
My first daughter is just in the process of weaning at the moment.
I kind of doubt that, because if baby is after term, the risk is supposed to be childbirth itself being too dangerous. And they induce birth if the baby is too big too.
Large baby is considered a risk on itself.
Since babies traditionally breast feed and thus survive solely on the mother’s energy after birth, this explanation doesn’t seem to hold water.
You have this a bit backwards- yes brain size at birth is limited by pelvis size, but this means that humans are born prematurely compared to other mammals. The entire body is basically premature and underdeveloped, not just the head/brain. If anything the brain is relatively over-developed, as it's size is the constraining factor in gestation length.
If I remember correctly, 18-24 months is the estimated gestation time humans would have if not restricted by pelvis size (which ironically is itself a result of walking upright), and this fits better with the idea of being able to walk closer to birth.
There's a chapter in "Born to Run" that ties human physiology and endurance and evolution and brain development together in a way that's so elegant, it almost has to be true. It covers the steps that lead to the need for a bigger brain and walking upright as a means for persistence hunting. Highly recommended reading, especially that chapter.
Or maybe having 4 legs (as a foal) gives more natural balance than a human or maybe even a cat or dog.
In the same way technically you only need one leg to walk (hop) around but that's much more difficult than having two.
However, another commenter brought up the fact that human muscles are somewhat underdeveloped after birth. It might make sense that having 4 legs is easier both in regards to balance, as well as because the load per leg decreases to make standing on them easier.
That is probably not the only reason, but probably a contributing factor.
It isn't. Human babies prioritize differently. We learn to understand language and to communicate before we learn to walk.
(In fact, for many babies learning to walk is a conscious process. We learn to learn before we learn to walk.)
Isn't it just a matter of priorities? Many birds spend a similar proportion of their life unable to fly, but they're safe in a nest. If you're a foal on the savanna, you need to get up.
https://www.stanfordchildrens.org/en/topic/default?id=newbor...
> Stepping reflex > > This reflex is also called the walking or dance reflex because a baby appears to take steps or dance when held upright with his or her feet touching a solid surface. This reflex lasts about 2 months.
Maybe not enough to qualify as "exhibit the basic motor skills"
https://www.smithsonianmag.com/smart-news/dragonfly-undertak...
Insects that have large migration patterns - all of the individuals die before coming back to the initial spot; only their descendants continue the cycle. How?!
In addition think about how bees can tell harvest locations to each other by dancing: https://en.wikipedia.org/wiki/Waggle_dance This dance is probably also something those bees know before being born.
Hence why the lottery ticket network (being a smaller version of the same initial network) can achieve similar accuracy (but not 'better') in much shorter time - because you took away all the confounding stuff. That kind of 'counterfactual' doesn't really translate to a biological domain, right?
Also - the baby brain is not really a 'pruned', smaller version of the adult brain, right? Neither is the fetus brain compared to the baby brain (which is more to the point of the article). So the analogy breaks down there too
I guess what the article describes is a bit like "pre-training" a full network with 'synthetic' experiences that quickly tune it for later. In that sense, I think its like these "dreams" are a 'distilled dataset' [1]. The question then is: how is this dataset being passed on from mother/father to child?
[1] https://arxiv.org/abs/1811.10959
Abstract:
> Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a small number of data points that do not need to come from the correct data distribution, but will, when given to the learning algorithm as training data, approximate the model trained on the original data. For example, we show that it is possible to compress 60,000 MNIST training images into just 10 synthetic distilled images (one per class) and achieve close to original performance with only a few gradient descent steps, given a fixed network initialization. We evaluate our method in various initialization settings and with different learning objectives. Experiments on multiple datasets show the advantage of our approach compared to alternative methods.
I think it is the other way around -- adult brain is a physically pruned version of baby brain. We start with lots of connections from everywhere to wherever and then lose a lot of them as we learn, while strengthening useful ones.
Or... has anyone tries training a network for determining the initialization state of another network? And now I'm wondering if those good starting initialization values only work well for a specific task or if they seem to span all or a subset of tasks, and maybe there's some inherent quality in how the values relate that we can tease out...
See "Weight Agnostic Neural Networks" (https://weightagnostic.github.io/):
> Inspired by precocial species in biology, we set out to search for neural net architectures that can already (sort of) perform various tasks even when they use random weight values.
In the days when Sussman was a novice Minsky once came to him as he sat
hacking at the PDP-6. "What are you doing?", asked Minsky. "I am
training a randomly wired neural net to play Tic-Tac-Toe." "Why is the
net wired randomly?", asked Minsky. "I do not want it to have any
preconceptions of how to play." Minsky shut his eyes. "Why do you
close your eyes?", Sussman asked his teacher. "So the room will be
empty." At that moment, Sussman was enlightened.I wouldn't compare this to dreaming really, it's a very different process where top-down connections hallucinate sensations. This is a completely bottom-up process, it's more or less like a test-bench connected to the optic nerve generating moving edges pre-birth.
"In a sense" ... A bit of a click-baity title for what amounts to suggesting that brain wiring is primed for sensing the world, which is not news.
The study itself looks to be more about how the brain primes itself (visual systems specifically). It may or may not have anything to do with dreaming.
What's even more fascinating is that emotions, behaviors, imagination and dreams must all be encoded and not learnt with just a billion-odd pairs / bits.
Meanwhile, it takes a few megabytes for us humans to encode a decent Hello World program in a modern programming environment.
conways game of life is derived from 4 simple rules yet shows complex behavior including the ability to operate a universal turring machine.
a few hundred megs of instructions seems quiet reasonably able to generate a highly complex system
Dreams help the brain in generalizing the world. From the article:
"That is, dreams are a biological mechanism for increasing generalizability via the creation of corrupted sensory inputs from stochastic activity across the hierarchy of neural structures."
Freud also predicted this in that he postulated that hallucination was more primordial than actual sense-experience
There was an example case of a blind man with cataracts getting it fixed at around 50 years old. People were curious if he would be able to know if a sphere was round just by looking at it without touching it.
IIRC - not only could he not do that, but he couldn't visually interpret shadows (saw them as black splotches - didn't recognize depth), and was confused why objects got smaller as they were moving away.
I think the human visual system trains on a lot of visual input data, but it typically happens at the baby stage where you can't really interact.
But I doubt it's 100% "no pre-wiring". What about gravity and musculature? Surely brains must know or train on these in some way before birth.
And what about fight or flight reflexes and presupposing monsters in the shadows? My understanding is that these had a primitive evolutionary basis and that we came hardwired to respond to certain stimuli.
The cataracts thing was an article I saw on HN a few years ago, but don’t remember the title.
Found this:
https://news.ycombinator.com/item?id=17687533
which leads to:
https://www.newyorker.com/tech/elements/people-cured-blindne...
which leads to:
https://www.newyorker.com/magazine/1993/05/10/to-see-and-not...
> TO SEE AND NOT SEE By Oliver Sacks May 2, 1993
> A NEUROLOGIST'S NOTEBOOK about Virgil (pseud.), who lost his eyesight as a child, and regained it at age 50. Tells how he could not adjust to the sighted world, and eventually had to be hospitalized.
Unfortunately, it needs a New Yorker account to view.
I don't think there's much opportunity to experience gravity before birth in order to train it, and I can't think of anything the brain would need to know about gravity beyond how we experience it with our senses.
> And what about fight or flight reflexes and presupposing monsters in the shadows? My understanding is that these had a primitive evolutionary basis and that we came hardwired to respond to certain stimuli.
Maybe pre-wired behaviors can depend on non-pre-wired stimuli, like a function that's conditioned on an undefined function and evaluates to false on exception.
Seeing differences at daylight or obscurity while in the womb does not seem impossible. Anybody that has closed the fist over a strong flashlight knows that the flesh is not totally opaque.
Is this dreaming? Maybe, maybe not. Other explanations are possible.
"Mammals dream about the world they are about to experience before they are born"
Wow, those are some evocative feels in those words.
I immediately thought about Google's "DeepDream" and how reversed back propagation gives us a projection of the network. Growing brains similarly fire before they're trained on real world perceptual inputs. Maybe these have shapes that evolved to be world-like and give babies an advantage in the fitness landscape.
My next thought: what if what we're collectively doing now -- all of our neural connectomes growing, dying, forming new connections, evolving understandings. Society, basically -- is the same shape of an AGI before birth? Maybe alien worlds tend towards this path too. A bad analogy, sure, but maybe it's the shape of the universe's consciousness being bootstrapped, before it awakens. Not getting into metaphysical garbage -- just thinking about what's to come.
I've heard the saying that fetuses aren't conscious, but I've never heard anyone, however fringe, say fetuses aren't alive.
I've never heard someone say farm animals can't feel pain.
> Eyes wide shut: How newborn mammals dream the world they’re entering
Reference to the Kubrick film I assume.
It's easy for me to believe that this could have initially evolved to give babies slightly more preparation for the outside world (and therefore a slightly higher chance of survival), and that we just happen to retain the ability throughout adulthood.
For those wondering how to check if you're dreaming: pinch your nose closed with your fingers. If you can inhale through closed nostrils, you're dreaming.
I dug a bit and was able to find this paper https://pubmed.ncbi.nlm.nih.gov/15289028/ , which seems to track my memory. It has a number of forward and backward citations if you are interested. Perhaps someone with far more than my vague recollection of a talk will step in though!
DOI: 10.1126/science.abd0830
It seems planaria worms can do something like that [0].
Some people today need to learn this.
Given that we don’t know what dreaming is, that analogy adds unnecessary confusion to interpreting the phenomenon described in TFA.