Study reveals how, when a synapse strengthens, its neighbors weaken | Hacker News Reader

Study reveals how, when a synapse strengthens, its neighbors weaken

news.mit.edu

194 pointslaurex41 comments

My naive take on this is that it makes biologically sense to keep net expected electrical impulses approximately the same before and after strengthening. At the end of the day the brain has energy constraints.

This can-be/is done functionally in ANNs but achieves a different end (avoids over-fitting) but doesn't reducing energy(compute) expenditure in dense ANNs since activation and non-activation is computed in expectation and take the same number of cycles in dense networks.

I'd love to see more work on massive sparse networks, where you actually get compute efficiency if you can reduce number of activation without reducing hurting your optimization target.

And apart from speeding things up, it could also have a regularizing effect.

In deep learning this is called local response normalization (LRN).

ConvNets use LRN where most active neurons inhibit other neurons at the same location in neighboring feature maps. http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf

Actually the new finding is similar to weight normalisation for individual neurons. LRN would be similar to another thing called lateral inhibition.

The naive conclusion is to avoid learning in any form in order to keep all the synapses in optimal state. Rather disappointing.

That's too naive. There's next to nothing known about how activity in individual neurons and their synapses relates to mental contents, in particular "higher level" concepts and thought patterns that you're concerned with in adult learning. In other words, the network dynamics are complex and unknown, and so the suppression / deactivation of certain synapses may just as well be a normal and necessary part of learning as it may be a part of forgetting, or neither. There is currently no contender for a "neuroscience standard model" that would bridge between this kind of neural dynamics and cognitive functions. I hope to live to see one.

Amen to that, we are starting to get some of the groundwork but are far far away from a “standard model”, neuroscientists,psychiatrists and AI developers oversell their understanding in order to keep their jobs and funding.

I know of no psychologist or neuroscientist who even try for a 'standard model' level of understanding of brain function.

It’s really just the AI folks.

Maximizing total synapse strength would as useful as maximizing the number of 1-bits on your hard drive. It's not what you want :)

Regardless of one’s general level of intelligence, there is that age-old adage of a trade-off between book smarts and street smarts, or any other niche skill. Forgetting, and the resulting ignorance, is part of learning. Perhaps that’s why people in modern societies feel unhappier than those in poorer, less developed societies.

Still, any society has certain forms of learning with outsized benefits that make them worthwhile.

I remember reading a paper a little while ago that stated that experts are better at their craft because they're better at ignoring unimportant and irrelevant input.

>Perhaps that’s why people in modern societies feel unhappier than those in poorer, less developed societies.

I have heard this before and would really question if it is true.

Has anyone studied this?

... and a model predicting and explaining this a decade ago: https://www.ncbi.nlm.nih.gov/pubmed/18602704

I wonder if the progress in artificial neural network technology is going to be correlated to discoveries on real neural networks.

I'm still wondering about the natural equivalent of back-propagation.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3673183/

Citations here:

https://scholar.google.com/scholar?cites=6395717759743355511

I think the term "backpropagation" has a different meaning in neurobiology.

those are backpropagating spikes, not errors

Isn't that mostly a supervised learning technique? Maybe the larger question is whether learning in the brain is better described as unsupervised or supervised.

I don't know how babies learn to initially use words but after that entails 99% supervised learning through school IMO nowadays. I would even argue that most new economic growth is the also the result of general supervised learning (aka, learning from others because the wealth of human knowledge is way more than anyone can learn in a million lifetimes and so the economic growth comes from continuously combining new ideas that combine other new ideas).

But we aren't beating kids when they get facts wrong and we aren't giving them candy when they are right. So studying how the reward signal is computed should be interesting.

Beating kids and giving them candy is way on the extreme side of reinforcement. Having good grades be praised by teachers/parents and bad grades be associated with disappointment is already pretty good since humans are highly receptive to other people's emotions.

Really we are being too static in the thinking about it. Supervised learning can essentially be done by one brain region supervising another that is being trained. It doesn't have to be outside the skull.

Yeah I agree! I'm just saying there's gotta be a lot of intricate machinery to make such subtle reinforcements work.

Here is the paper: http://science.sciencemag.org/content/360/6395/1349

So this is how Eg brainwashing works? Repeating a message strengthens the new pathways until the old connections are too weak to fire?

I would think any kind of training would have this effect. In mechanism—not purpose or ethics—brainwashing doesn’t strike me as all that different from ordinary practice.

Arguably, a single message would only affect a single pathway.

AFAIK a single message causes a giant electrical and chemical thunderstorm in the brain, not only one excited single pathway?

You are right of course. But because a trivial message has many "layers" of "messages", according to semantic theory, it's often a single lie in a whole message packet.

So basically L2 regularization.

Not really. This is a local inhibition, and it's not clear if there's penalty on synapse growth.

How interesting.

This looks like a lateral inhibition effect running on biochemical means that are persistent compared to the transient electrochemical phenomena of classic neuronal lateral inhibition. The latter does lots of signal processing in visual and other systems. [1]

[1] https://en.wikipedia.org/wiki/Lateral_inhibition

We are born with a ton of synapses as babies and through a process called pruning, many synapses die off, unused branches.

This seems pretty normal to me

Really curious what the biology-centric AI folks have to say about this, e.g. Jeff Hawkins.

Normalization of a layer’s outputs kind of does this already.

So it's similar to "lateral inhibition", but for synapses.

Isn't that kinda the point? It's called learning.

Does this mean addiction can completely be cured

self pruning mechanism ?

Reply on news.ycombinator.com