Artificial brains may need sleep too

The model that they are creating is a sparse model. This means that, for the vast majority of possible inputs, the machine should produce no hypothesis whatsoever about what it is looking at. In fact, a good sparse representation only produces a representation when the input is in the space of valid inputs. If the model is given a valid input, it should come up with a representation of that input, if it is given garbage, it should prefer to not try.

Signal is pretty easy to find, you just open your eyes, explore your environment, and update your weights accordingly.

We would alternatively like to train the model to recognize when it is looking at nothing, but we have to be pretty careful about what we show it. If any signal is present at all, we definitely don't want to discourage our model from finding it. That was, after all, the point of the model in the first place. Fortunately, because of the size of the space, we can be certain that if we completely shut off the inputs and feed it noise instead, it is very very far from a valid input, and anything that it includes in its representation was included in error.

> “The issue of how to keep learning systems from becoming unstable really only arises when attempting to utilize biologically realistic, spiking neuromorphic processors or when trying to understand biology itself,” said Los Alamos computer scientist and study coauthor Garrett Kenyon. “The vast majority of machine learning, deep learning, and AI researchers never encounter this issue because in the very artificial systems they study they have the luxury of performing global mathematical operations that have the effect of regulating the overall dynamical gain of the system.”

2) Well, it would get in the way of actually living? If you're talking down some stairs and then your brain's motor cortex starts working on consolidating a skill and you fall you'd hurt yourself.

3) I don't think we know that. Possibly we had to sleep for memory related reasons and our bodies just evolved to use that time for other sorts of important housekeeping when it wasn't moving around doing stuff?

4) Learning is very important and takes time. In addition to the procedural memory consolidation in REM sleep we also move long term memory to a more stable form of storage in other phases. But neural nets don't really have anything analogous to a brain's explicit memory so this doesn't really bare on that.

The why we sleep book made me think of this hypothesis.

He states that dreaming likely came before consciousness too which I found interesting.

As in, our early brains form reality using the same concept as dreaming.

So I’d argue that ML systems already “sleep”, it just looks more like dolphin sleep due to their similar need for continuous operation.

Definitely not as some sort of half baked ML project though that pushes simple articles on demographics it thinks can't read.

You're proposing something like "Accept-Semantic-Complexity", or maybe extend Accept-Language somehow to include acceptable complexity. It could work in principle but I doubt anyone would use it. Many websites already have a tendency to ignore Accept-Language and instead use some fuzzy region detection when they offer multiple languages.

That said, expandable context and background boxes are still a great idea.

Think about how real-time guarantees on GC are bad for throughput.

Evolution can get "stuck" when optomizations become obligatory. One theory of "hot bloodedness" (which is not one thing, nor something that evolved once, to be clear) is large species that passively maintained some body temperature then shrinking and being forced to actively maintain it.

It is a simple, easy to state, hypothesis of what sleep is doing in organic brains, and you should note that there is an extreme paucity of those.

Perhaps the artifical brain referred to in the article is not the kind of artifical brain you're used to reasoning about, but the goal of these researchers is not to optimize their performance on ImageNet, it is to discover how the brain actually organizes itself. You should give it a read.

“AI may need training too!”

Machinery does not need to sleep. Networks do not need to sleep. Roads do not need to sleep.

Living creatures do.

Machines and Living creatures are not the same thing, they do 'learn' the same way.

A neuromorphic processor is a processor that tries to function as the brain does. I say tries to because the models are still very simplified versions of how we think that a neuron works. The spiking part comes from one of these models. A neuron will only send an electrical signal to another neuron once a certain threshold is met, this is what is meant by a spike. What this allows us to do is to add a temporal/time component to a neural network. You've probably heard of "neurons that wire together, fire together". If you have a lower threshold it means that the neuron you are sending the signal to is more relevant for whatever thought process is going on right now and vice versa. New input can change these thresholds.

The biggest promise of spiking neuromorphic computing seems to be a massive reduction in energy usage while still offering decent accuracy. So for example you could use it to train a neural network to get to 80% accuracy after which you'd let another type take over to get to 95%. This field is still in its infancy though, so expect things to change/improve fast.

Modern airplanes don't flap their wings either.