Comment by smallmancontrov | Hacker News Reader

Comment by smallmancontrov | Hacker News Reader

Parent Full thread

smallmancontrov

Sleep isn't about cleaning. It might include cleaning, but I don't believe for a second that cleaning is the main function.

Why? Cleaning is easy to optimize yet sleep is evidently hard to optimize. Going unconscious for a big chunk of the day in a world where everything wants to eat you is a crazy price to pay, yet sleep is highly conserved across species under this extreme selective pressure. Even the animals famous for avoiding sleep don't really avoid sleep so much as bend the rules! Seals partially wake up to breathe, migrating birds sleep a hemisphere of their brain at a time, but they all sleep. This didn't happen because "pumps are slow" or "filtering is hard." Biology is excellent at copy-paste-reinventing that stuff. Optimizing the pumps and filters is not something evolution would have struggled to do if it could have meant not going unconscious for a third of the day. Especially for organisms with small brains where the transport logistics are trivial.

IMO the much more likely explanation is that the brain uses contrastive learning somewhere. We know animal brains don't use backpropagation to train and contrastive learning is one of the biggest competing family of techniques. Instead of using calculus and outer products to calculate derivatives, you feed real data and false data forward through the network and each neuron tracks and trains on the difference. This can be batched -- many CL techniques could handle 16 hours of real data followed by 8 hours of false data -- but the false data can't be put off forever, or the contrast signal goes away. Also, you probably want to inactivate I/O while the false data is flowing through. In other words: sleep emerges from first-principles on the category of neural network learning techniques that the brain probably uses.

I don't think anyone was seriously about to shoot up with sleep-replacement peptides, but also: don't. Let someone else figure this out the hard way.

I think you overestimate the disadvantages of sleep. It's very species dependent, for some it might be a disadvantage because you're more vulnerable to predators, for others it's an advantage because you spend less energy. If you're safe you really only need enough wake time to eat and reproduce.

If you're safe.

You definitely see a bit of this in other primates. They sleep on trees (much safer) and for a lot more hours than humans.

That's great for primates, but what about nematodes? Everything with a nervous system sleeps, even when the cost is high.

Obviously sleep is necessary. I'm saying that there isn't always a large pressure to minimize sleep. Long sleep can be beneficial just to preserve energy.

Right, but my argument is "high pressure never managed to push past this barrier, therefore it's a strong barrier." The existence of other, lower pressure circumstances doesn't have any bearing on the conclusion -- unless you want to argue that the low pressure circumstance is so universal that the high pressure circumstance doesn't exist, that no animal anywhere is inconvenienced by the need to sleep. But I don't think that is what you were going for.

Sleep has to be essential to life as we know it, because no animal has completely outevolved it. Some animals sleep an hemisphere at a time (dolphins, long-flight birds) or in fragments (giraffes), but none truly goes without sleeping no matter how challenging their environment is for a nap.

> Cleaning is easy to optimize

Your entire argument rests on this statement, yet I don't see any further talk about it or anything to back it up. Perhaps sleep is the very best way to optimise cleaning, outweighing the disadvantages?

Biology does an insane amount of packing and shipping in these spaces already and this is the stick against which we should measure potential modifications. While a neuron is active, vesicles will be riding up and down the axon tubules, shepherding RNA and building blocks, releasing and reuptaking neurotransmitters, and dozens of ion-selective transports create the electrical behavior. At the cellular level, the brain has its own immune system, at the tissue level it has the blood-brain barrier. This is all neuron-specific innovation, everything from basic cell biology still applies. The point is: if you are familiar with the biology, you are familiar with a dozen transport systems (and can infer the likely existence of a hundred mechanisms) more sophisticated than waste evacuation and none of them require the neuron to go offline, so the prospect that this is the hangup seems unlikely.

Also, the complexity of transport scales with the system. Famously, zebrafish don't need a functioning heart for the first week of their life because at that stage they are small enough that diffusion does a good enough job transporting nutrients and oxygen. Needless to say, this goes for waste evacuation as well: small size trivializes the problem. Nematodes are even smaller that zebrafish, yet they still sleep.

"Sleep is about training neuron functionality" explains the universality of sleep much better than "cleaning is hard."

If that were true, wouldn't you expect 50 year olds (who have mostly learned how to navigate life) to need little or no sleep? Seniors sleep only a little less than young adults, still 6-8 hours a night.

There are other known mechanisms in sleep, for example copying/condensing short-term to long-term memory by replaying it. Translating increased compression into better fundamental performance seems like a steep task and 50 year olds have already reproduced or not, so shrug

I think we made a big mistake by using biological terms for computer processes and now we have people like you coming to the """natural""" conclusion of going full circle and applying computer terms back to biological processes.

There is virtually no parrelel between both worlds besides the naming and no one has any idea about what's going on for real.

I keep hearing that. At the same time people makes comments like the one you just answered to, using that analogy heavily. Could someone who understands both say where the analogy breaks down in this case?

The brain is...

~a machine~

~a steam engine~

~a computer~

an LLM

Analogies help understand the brain but OP is taking the analogy and then going back and applying principles from the analog to the brain.

The brain is the brain, it isn't any of those other things.

I don't see a need to be dismissive. There is a lot of evidence that there is a relationship betwwen sleep and learning. The hypothesis advanced by the OP is worth evaluating on its merits rather than being dismissed as a fale equivalence.

This smells a bit like a statement of faith. Digital neurons and neural networks are not the same as their biological counterparts, sure. But it is not an unreasonable proposition that their functions are isomorphic.

> not an unreasonable

Yet we have absolutely 0 clue nor proof besides "it feels about right", that's not how the scientific method works.

It's not zero clue. Neurologists have got a lot of observations on how the brain works and there are multiple plausible proposed mechanisms for how the brain might do something like backpropagation, it's just that the details are really messy so there's not a single clear picture that has emerged.

(Similarly, there is decent evidence that the brain uses higher-dimensional vector representations of concepts in a similar way to neural nets, at least in some parts of it, but again the details are messy and there's not a clear idea of how these representations emerge or how to decode them reliably)

It is unreasonable, because a neuron is categorical different from a so called "neuron" used by AI.

Same is true when you compare an actual brain and an AI model.

While there is much to suggest that sleep is important for learning, why contrastive learning in particular? It seems to me that the "deactive IO" argument could apply to any learning technique. Similar to how railroad maintenance is carried out at night because it is too unsafe to do during the day when trains are whizzing by.

Contrastive learning provides an unusually strong push towards an offline solution, even though the offline solution is expensive.

Road maintenance isn't a good analogy; it _can_ be done "online" (with traffic, during the day) and in fact _is_ often done online, despite the pressure to do it online being less than "road workers will be eaten by monsters if they venture out between 8pm and 6am." This is the opposite circumstance we see with sleep. Animals always sleep despite the steep cost. If we want to fix the analogy, we could imagine a world in which all roadwork was done between 8pm and 6am despite workers regularly getting eaten by black-furred creatures of the night. Wouldn't that beg the question "why not do it during the day?" If you learned that every city and state and country did the same crazy overnight work schedules despite losing workers to them, wouldn't you suspect that there was a strong hidden reason?

Ok, but this has nothing to do with contrastive learning. Your argument applies to all kinds of self-supervised learning methods.

However, the contrastive learning hypothesis also explains dreams and why they make no sense.

I think it's often a bit of a mistake to assign one function to a given feature of a biological system. Biology tends to be messy and tangled and so something can quite easily have multiple functions.

Sleep is literally cleaning. It is flushing out the brain.

Cleaning happening during sleep doesn't mean it's the main purpose. It could be merely a convenient time for doing the majority of it.

dreams are false data

exactly!

*synthetic training data

> the much more likely explanation is that the brain uses contrastive learning somewhere

This doesn’t explain sleep in virtually brainless creatures. A metabolic root cause seems more likely.

Why would neurons be exempt from training just because they weren't clustered into a brain? By comparison, transport-related phenomena are almost completely trivialized in the simple creatures. The existence of sleep in simple critters seems strong evidence in favor of contrastive learning over cleaning explanations.

Could you make the case for metabolic explanations? Are there a bunch of coupled mechanisms that all require downtime and couldn't possibly be phased?

> Could you make the case for metabolic explanations?

Animals without nervous systems exhibit sleep-like states [1]. Potentially even animals without neurons [2].

[1] https://pubmed.ncbi.nlm.nih.gov/28943083/

[2] https://www.science.org/content/article/if-alive-sleeps-brai...

Reply on news.ycombinator.com