Brain bursts can mimic famous AI learning strategy
quantamagazine.org
quantamagazine.org
Counterpoint: Our brain works around "lapses in our vision" constantly. See https://en.wikipedia.org/wiki/Chronostasis or https://en.wikipedia.org/wiki/Blind_spot.
(Not saying that makes backpropagation biologically plausible, but as phrased, their argument is not convincing.)
ie., our visual field lags during, eg., a saccade; but all the neurones backing the visual system are still responsive to light. And it isnt that the visual field is blank, nor the eye/neurones unresponsive.
The neurones are in constant continuous contact with environmental stimuli -- they don't just stop being causally activate.
The argument given in the article is very much that you would perceive an interruption if your neurons were momentarily doing something else, but you don't, so they don't ("A -> B, not B, therefore not A"). I'm contesting this implication. You merely presented statements equivalent to "neurons don't do that" ("not A"), which misses the point.
Or, phrased another way:
How do you know that "[neurons] don't just stop being causally activate"?
It is not inherently implausible that, at any given time, some neurons in our brain are "on backpropagation maintenance", so to speak. ANNs can be trained just fine with dropout after all (and for ANNs, we intentionally add dropout because it's beneficial!). I have no evidence for this claim (and I don't believe it to be true in the slightest), but "you'd notice a momentary loss of perception" is simply not an argument against it.
At any rate, the degree to which neural networks are modelled on neurons, despite the name, is vastly overstated. A neuron and a ReLU are very different beasts.
ReLUs specifically, IIRC were made after it was realized that to satisfy Kolmogorov theorem, any nonlinearity would suffice, so ReLUs were chosen due to their computational simplicity.
Convnets were not inspired by biology directly and the only property they share in common is a spatial receptive field. The weight sharing and overall architecture were not based on the visual cortex.
ReLUs were chosen because of their constant derivative in the positive domain. The reduced compute was a nice side benefit and newer methods (swish, gelu, mish, etc.) all do away with the simple computation in favor of smoother loss landscapes (while maintaining a mostly linear positive domain).
Backprop, however, takes no inspiration from biology.
Yeap that's what I meant by inspired. It's a pretty weak inspiration.
Thanks for the correction on ReLUs.
Actually IIRC also the important bit about ReLUs is that their range is unbounded so that the derivatives don't diminish to zero over multiple iterations over the activated path.
However as soon as you make it a supervised optimisation alg it fails even to be a model. (The NN algorithm is just k-nearest-neighbours + recursive PCA.)
Activation patterns in the brain aren't "trained" by backprop, and they dont form a piece-wise linear (ie., ReLU) regression function.
It's better to say that the optimisation algorithm called "Neural Networks" was inspired by the "Neural Network" model of the brain which was a partially highly simplified model of one aspect of neurone behaviour.
The algorithm used today has nothing homologous with anything in the brain -- and its somewhat incoherent to suppose it does. A piece-wise linear regression model of the boston house price dataset is a meaningless stand-in for anything neurological. Its aburdist marketing to say otherwise.
Agreed, but the principle of convergent evolution is a thing. It's possible that and we cannot rule out that we are accidentally stumbling our way into homologous models of biological compute (I personally doubt it).
it is possible that the activations in a NN might instantiate this function... but that's literally probability 0.
As the set of such functions, R^2 -> R are uncountable infinite.
The modern algorithm called "Neural Networks" has nothing to do with the 1940s model of the brain called the "Neural Network" model.
The modern algorithm is a supervised optimisation process which approximates functions using a piece-wise linear regression model. Nothing about that algorithm, nor the function-approximations it produces, is based on anything in the brain.
It's a bit like saying the Tesla was inspired by horses.
The process of generating a NN is it’s own thing, just as python code is different from a python programmer.
Neural networks are only in the loosest sense inspired by biology. Backprop itself was not inspired by biology. The implementation of backprop has been known for a while to not be biologically plausible for a number of reasons, some covered in the article. Even the paper’s burst method isn’t backprop. It’s a mechanism for error transport. Errors themselves are still localized which is a big break from backprop. Thus “mimicking” is correct because only the outcome is similar, not the mechanism.
actual title: Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits
It is a fascinating poem -- it all takes place in the character's head. The virtues and the vices battle it out, individually, but as a complete battle. It is, in my opinion, a great example of a Generated Adversarial Network.
The reason a lot of great art ends up representing deep mathematical truths is that art is, at its best, an act of deep observation, whether it is of oneself or the world around oneself.
I wouldn't bother reading the Psychomachia -- I translated parts of it in college and I agree with C.S. Lewis that it is a more known poem than a good one. I still found its observation of the mind in parts doing battle useful for myself when I have dealt with mental illness in my life.