Comment by bbctol | Hacker News Reader

Comment by bbctol | Hacker News Reader

Parent Full thread

RELU is decidedly the more "biologically accurate" (or at least biologically inspired) compared to the sigmoid function. The idea put forth to originally justify RELUs was that neurons in the brain only have action potentials or lack thereof: the lowest level of activity a neuron can have is zero, and then you can ramp up frequency of action potentials from there, as opposed to the sigmoid's positive and negative symmetric curves.

That said, a lot of the history of neural networks has been brief moments of biological inspiration followed by hacking and engineering that drifts further away from the biology the better it gets. The biggest example is backpropagation; despite how essential it's been to artificial neural networks, it really doesn't exist in the brain, at least not as simply as it does in code. For now, we're all still exploring, some looking towards biology, some towards abstract principles, and it remains to be seen if one provides consistently better results.

> The biggest example is backpropagation; despite how essential it's been to artificial neural networks, it really doesn't exist in the brain, at least not as simply as it does in code.

Do you have any links to papers or such explaining "at least not as simply as it does in code"?

EDIT: nvm, I followed the links in a wikipedia article on RELU to relevant PDFs...

For a while now this is one area I have been questioning - that we do use backprop, and maybe there is something to be learned from nature that might (?) simplify how a NN is trained (then again, nature might be doing it in such a way that is more complex than can be engineered or practical)...

> For now, we're all still exploring, some looking towards biology, some towards abstract principles, and it remains to be seen if one provides consistently better results.

It might end up being a combination; at least, that seems the direction so far to a point.

I want to thank you for your comments, though. I'm still learning this stuff (I'm working thru the Udacity Self-Driving Car Engineer Nanodegree), and you've given me some stuff to think about and explore further.

Honestly, the current state of neural backpropagation research is still in flux and papers will be pretty technical. My background's in neuroscience, and the rough summary of what we know so far is: action potentials can travel backwards down a neuron, and they probably do so in vivo, but we don't know if this actually does anything. There's no mechanism we understand for passing information backwards through a synapse. In fact, the only reason people suspect this might be important, and not just an accidental effect, is because backpropagation has been important in developing artificial neural networks.

Getting inspiration from nature is often useful (convolutional neural networks are loosely inspired by the way information is processed down the optic nerve) but for the specifics of neuron function, the brain is almost certainly doing things that are more complex than is practical to simulate. The brain contains hundreds of billions of neurons, and each one is orders of magnitude more complex than a node in an ANN, interacting with local and distant neurons through all kinds of separate but interlocking electrical, chemical, and long-term genetic actions.

Regarding "brain-like" propagation strategies:

Is there any NN model which tracks a "supply" each neuron has of signalling "molecules", such that a given neuron won't be able to communicate a message of class X if it's out of X-amine, unless/until it receives some from a neighbour?

Or, is there any NN model that allows each neuron to send broadcast messages, ala extracellular ionic neurotransmission, which decay with "physical distance" (requiring each node on the neural graph to have a physical position) and which remain active until "sucked up" by something?

I've always thought these two things—neurons needing to "spend" chemicals for neurotransmission, and neurons being able to observe the local-in-physical-space "loudness" of certain broadcast signals—seemed to have high potential for modelling agentive systems generally, since they seem to replicate other successful systems (markets; pheromonal communication), but I've never heard of either concept being studied in an ML context.

The most mainstream of the "biologically-inspired" Artificial Neural Network variants is the Spiking Neural Network model, which IBM has been implementing in their TrueNorth chip architecture[0], and APT implements in their SpiNNaker[1] design.

That said, this is only from the perspective of constructing ANN's for computational purposes. There are, of course, many detailed models of various processes in the brain, but this is generally under the heading of Computational Neuroscience[2] (though the field boundaries are blurry). The main software for simulating biologically accurate neural networks is "Neuron"[3], which is used in the Blue Brain Project[4].

[0] https://en.wikipedia.org/wiki/TrueNorth

[1] https://en.wikipedia.org/wiki/SpiNNaker

[2] https://en.wikipedia.org/wiki/Computational_neuroscience

[3] https://en.wikipedia.org/wiki/Neuron_(software)

[4] https://en.wikipedia.org/wiki/Blue_Brain_Project

Would an action potential traveling backwards be able to block the neuron from properly firing, briefly? In other words, could this effectively act as a short term inhibition?

Thank you for this explanation - I would be curious about any papers, no matter how technical. I've never let something like that stop me before...

Nature uses something like STDP (spike timing dependent plasticity), where the change in a synapse strength is proportional to the sign and delta time of the last spike at each end of the synapse. You can approximate something like backprop by using symmetric STDP (discarding the sign).

"Towards a Biologically Plausible Backprop" from Benjamin Scellier and Yoshua Bengio (2016) would be a recent paper on the topic.

> "Towards a Biologically Plausible Backprop" from Benjamin Scellier and Yoshua Bengio (2016) would be a recent paper on the topic.

Thank you for this link; I've found a copy of this paper - and some others - and will be reading them with interest!

Reply on news.ycombinator.com