AI frenzy almost convinced me that sleep was the training of our neural network with all the prompts of the day.
And now this /o\
And now this /o\
Sleep itself is characterized by coherent neural activity— the large number of brain regions with synchronized neural activity. The slow waves where huge numbers are all firing close together in a rhythm. Low frequency and high amplitude delta brainwaves (1-2 hertz).
Complex adaptive brain activity requires more complex firing than a simple rhythmic frequency. So, in a way, the complex activity must be stopped in order to support global synchrony.
Why would our neurons want to all fire synchronously? Well, it is healthy for neurons to fire together in a causal manner— neurons release growth hormones then. That neural growth during synchronized firing is the basis of “neurons that fire together wire together.” And it doesn’t seem coincidental that a successfully predicted model feels good, as in the case of successfully throw a ball in a basket. Neurons are trying to predict other neuron firing and respond to it. If they are unable to effectively, they may become like the 1/3 of our baby neurons in the cortex — they will be pruned and die.
Good feelings is positive reinforcement—behaviors leading to good feelings get reinforcement. The feeling of harmony or harmonization, where we have to balance a broad set of internal neural impulses, feels good when we do it well. We feel harmony in music — and in our own internal sensory resonance to the world.
Hypothesis 1: the harmonization of neural activity might cause conscious feelings due to the convergence of the activity to platonic forms (see Platonic Representation Hypothesis in LLM research).
Returning to sleep — this is a proposal for why sleep feels good. Synchronization might intrinsically feel good. But because the sleep also disrupts your working memory contextual attunements (ie, whatever your day was about) - taking your brain into deep synchrony — it strengthens the overall dendritic connections between the synchronizing neurons.
And, because it wears off the edges of your previous experiences — you can return refreshed.
In this way, sleep seems to contribute to the overall integrity of the operation of our intelligence. Without it, we lose integrity and internal harmony.
And yet, not sleeping is one of my favorite drugs. Can be a major performance enhancer, even if it is variable.
Hypothesis 2: Not sleeping increases the (statistical) temperature of the brain.
Sleep deprivation is a well known treatment for depression [1]. Maybe you lean to the depressive side, that would explain positive effects.
[1] https://en.wikipedia.org/wiki/Sleep_deprivation#Treating_dep...
Periods of sleep certainly seem to be used in that sort of way, but that is an extra use evolution found for the sleep cycle once it existed rather than the reason sleep developed in the first place.
There are a number of things that seem tied to, or at least aligned with, our wake/sleep cycle that likely didn't exist when sleep first came about.
It's also a fundamental misunderstanding of how LLMs work, mixing up inference with training.
Can we say that after ChatGPT's release in 2022, now antitech bros think everything is about LLMs specifically?
Prompts are specific to LLMs. Most neural networks don't have prompts.
Additionally, prompts happen during LLM inference, not LLM training. There are many non-technical people who claim they have experience "training" LLMs, when they are just an end user who added a lot of tokens to the context window during inference.
It is pretty common during the fine-tuning phase.
It's like someone said while driving the car "let's give it some gas" and you said "but the tank is almost full" when they obviously meant "let's press the accelerator pedal"
Since in-context learning is a thing, “adding tokens to the context window”, at least with the intent and effect of having a particular impact on capabilities when inference is run on the context to which they were added, is, arguably, a kind of training.
Descartes compared the human mind to waterworks and hydraulic machines, other authors used mechanical clocks, telegraph systems, digital computers, and (in the recent decades) neural networks.
In the end it's all computing and to a degree all of those models serve as good analogies to the wetware, one just needs to avoid drawing wild conclusions from it.
I'm sure there will be new analogies in the future as our tech progresses.
We don't literally train on today's prompts while we sleep, but there actually _are_ some _computing_ tasks going on in our brains at that time that seem to be important for the system.