Hacker News Reader

I am with you on the latest hype in academia and industry of getting on the Deep Learning train and not being left behind.

In this particular article, the authors propose a hypothesis that is drawn from similarity of over-fitting phenomenon seen in neural networks to over-fitting in human brain and sleep as a noise-additive approach to prevent this overfit.

In order to judge this hypothesis, I would urge you to counter their arguments (esp sections 3.1 and 3.2) with what you find is misleading or incomplete or outright ridiculous. That way we can get to know both sides of the story.

I'm not the original commenter, but I feel similarly. Perhaps offering some thoughts on sections 3.1 and 3.2 could foster some discussion.

Essentially, the hypothesis is that dreams are some kind of regularization that the brain performs on learning to prevent overfitting. The authors propose a couple of different arguments for why this may be useful (from deep learning) and why dreams in particular is where this regularization might happen (from neuroscience).

I don't think anyone would object that overfitting is bad and regularization is useful. I might even agree with the author that some kind of anti-Hebbian or noise corruption may be happening in the brain to help with this. However, the implied analogies in this paper between the tricks used in deep learning and dreams feels really "just so" to me.

I think their main evidence is that:

  1. Dreams facilitate learning
  2. The dreams we experience generally reflect the repeated task we experienced during the day
  3. We often hallucinate variations of environments, not the exact environment we experienced

I think overall from this, I feel it is fair to say that the brain does something at night to help improve learning. It's not immediately clear that this is analogous to dropout, training augmentation, or generative models used in deep learning. It is even less clear whether dreams are truly needed to perform this. The authors don't really dissociate dreaming from sleeping in most of the literature that they cite. Remember that your experience and the weight changes in your brain may not be that highly correlated. Many neural changes may happen during a dream which are not correlated with your conscious perceptions/recollection of that dream.

Human, animal, and machine learning processes to have common points (vide https://p.migdal.pl/2019/07/15/human-machine-learning-motiva...), even if the low-level mechanics is different (at least for biological brains vs GPU operations). We were already puzzled by some similarities (see: "Does AI have a dirty mind, too?" https://medium.com/@marekkcichy/does-ai-have-a-dirty-mind-to...).

> Unfalsifiable non-insights

By all means, it is falsifiable. We can present some training materials and alter the sleep pattern for a fraction of subjects. If generalization is not more affected than the ability to work with the memorized material, we falsified that.

> We don't even know how the brain does almost anything and you think you can use trendy AI topics to explain something as complex and mysterious as dreams?

With this approach, we can safely quit all science. All in all, it is hubris that a mammal brain can understand the universe.

Insights, ideas, and testable hypotheses offer us a way to make educated guesses. Occasionally they provide a way to understand more.

You can think of DL as a model to explore huge solution spaces, is clearly not the only one but looking at things with some models sometimes makes much more sense and DL has been extremely useful. Is true than real neurons and artificial ones are different but "both are systems that perform complex tasks via the updating of weights within an astronomically large parameter space.". The brain is clearly not using the same mechanisms but it probably have some of the same problems and looking it from a DL perspective (that we understand better than the brain) could help us understand it better.

>Unfalsifiable non-insights by people trying to apply arbitrary deep learning algorithms to a brain which is definitely not using any of them. DLcentrism? We don't even know how the brain does almost anything and you think you can use trendy AI topics to explain something as complex and mysterious as dreams?

Mistaking analogies for models.

Analogies (e.g., "machine learning" and "neural network") can help introduce ideas in a discussion. And if the analogy sparks new questions, that's also fine. One just needs to come up with a valid model and run experiments.

But some people try to harvest new insights without leaving the analogy. This is worsened when they do experiments to find supporting evidence instead of trying to disprove the hypotheses.

The answer I got from Matt Walker's "Why We Sleep?" was a list of benefits that we get from sleep and a list of negatives that we avoid. That is a sufficient answer for me. To ask why we need those benefits is a different question and eventually goes down a philosophical path.

Personally, I think that most behaviours/attributes have evolved out of the basic laws of life (survive and reproduce) and of Physics, especially energy-related constraints. Based on that the most obvious basis for sleep is energy saving.

I believe that 'sleep' has been observed in animals as simple as jellyfish so any neuroscience-centred explanation is bound to miss the root cause, in my view. But, as often in evolution, the development of nervous systems may have taken advantage of pre-existing sleep patterns.

'Why We Sleep' has some, uh, issues on the benefits-and-negatives front too.

https://guzey.com/books/why-we-sleep

Thanks for posting - it's always good to see all views. It's disappointing because the book helped me a lot. It helped me understand sleep and how I should change my lifestyle to get more of it. I feel (no measure) that it has benefited me.

I will not regard it as scientifically accurate now though.

Does anyone have any other books about sleep that they could recommend?

As someone working at the intersection of neuroscience and machine learning: thank you, very much. You might enjoy this book we use as a bible in our lab: http://cognet.mit.edu/book/principles-of-neural-design

Putting it out there as an idea, might inspire others and might help.

Machine Learning is the most closest thing we have, as far as i know.

> Putting it out there as an idea, might inspire others and might help.

Yeah I agree, maybe it's broscience but maybe it will spark something interesting and to me that's worth trying and sharing.

That said, to be honest I didn't read past the abstract as I don't have the time but cool idea :)

>ridiculous hubris

>definitely not using any of them

Aha, this was subtle. Nice catch of the poster's own "ridiculous hubris" in assuming that the brain is "definitely not using any" algorithms.

When complex mechanical machines were big, we imagined pneumatic brains. Then we imagined electronic, digital, and now quantum computing brains. We also started to see the brain through the lens of these systems and how they work.

Meanwhile the brain continues to work however the brain works.

Once people talked about "iron horses". Now we have walking robots. From a current perspective, trains are nothing like horses; does that mean that it's invalid to say designing and making walking robots does require/provide insight into the physiology of legs?

It seems to me you're making a very general statement that there is no progress in understanding because people used to vastly overestimate their understanding. People did not know stuff, and therefore they will never know stuff, and therefore they do not know stuff.

> does that mean that it's invalid to say designing and making walking robots does require/provide insight into the physiology of legs?

Yes. That sounds invalid.

That sounds overly binary to me. (said with conscious irony)

You are saying that the informational inputs and outputs for walking robots have exactly the same amount of relevance to biological limbs and biological control of walking as the engineering of steam locomotives?

Obviously they have aspects that are nothing like walking creatures. But the more they solve the same problems, the more they have to be similar, either intentionally or accidentally.

I'm not arguing that our understanding hasn't advanced, only that our models are constrained by our internal conceptual and external verbal/written vocabulary. It's possible that brains are doing things that we have no thoughts or words for because they don't resemble any system we've ever dealt with.

They're just advancing a hypothesis. If you read the paper you'll find it is written quite humbly, makes a good case for why the hypothesis is not crazy, and makes testable predictions which could falsify the hypothesis.

No downvotes. I agree with you.

I can guarantee you that deep NN/RL researchers are thinking of these hypotheses every other week. But they don't publish them. Even I, who got more active in this kind of research very recently, came up with the exact same hypothesis. I didn't publish.

We don't publish because we have a strong sense of empiricism. There proverbial proof has to be in the pudding. We should be able to setup a DNN and run experiments. If it confirms our hypothesis, we publish. If it disproves our hypothesis, we move on to something else.

These armchair neuroscientists/psychologists come along and type out a dozen pages of brain-dump, no pun, and think they're doing science. They need to do better than that, and do some real work with hands-on deep learning, and/or experimental neuroscience (or collaborate with someone who can do that).

This comment on its own can be read as testimony that covid-19 self isolation / social distancing can cause a compulsive need to try to feel relevant in some way when you meet to little people in situations where you are actually relevant. So yeah, I know it is somewhat preposterous to claim any relevancy, but one has to try to cope in some way.

I'm only just "DNN-literate" enough to get the broader points of the more well written papers. I certainly don't know enough to give any criticism on my own, but in a somewhat funny way, statistics, and my experience could perhaps lend some additional weight to your argument.

I'm pretty sure this is very similar to one hypotheses I've amused myself with from time to time, if not exactly. I don't even think I knew what the word overfitting meant at the point I came up with it. Though the ideas involved would seem to boil down to the same thing, or close enough to gauge how common this line of thinking would be.

It's a sample of one, but since I don't believe myself to be that special, and don't believe others would believe so either, it adds some weight that this line of thinking might be relatively common among anyone with even a little knowledge in the field(s).

Perhaps one could even argue I ought to be relatively bad at coming up with novel (creative) ideas, and ideas which are not precluded by known facts. At least partially because my brain would have a higher chance of being overfitted to the few bits I know. Any of my ideas should then either be shallow or false with high probability.

There are way way worse people out there than the ones who put up curious papers on a pre-print server for others to read for free.

> There should be a name for this kind of ridiculous hubris

Pseudoscience? To me this paper is no different from astrology.

> There should be a name for this kind of ridiculous hubris. Unfalsifiable non-insights by people trying to apply arbitrary deep learning algorithms to a brain which is definitely not using any of them.

Galaxy brain-ism