Comment by cr0sh | Hacker News Reader

Comment by cr0sh | Hacker News Reader

Parent Full thread

Something I'm finding interesting is how so many of these deep-learning neural nets use RELU for their activation function; RELU is known as a "lazy engineer's activation" function - very simple to implement, and despite looking like a hack, seems to work very well for many tasks.

I tend to wonder - beyond "ease of implementation and good 'nuff" reasons - if there are other reasons to use RELU, over other activation functions like TANH or Sigmoid?

I'm beginning to suspect that we may be seeing the "engineering side" of neural networks coming into play; that instead of using the more "biologically accurate" activation of the sigmoid function, we instead use RELU (and other ELU derivatives) because it works well, and is easier to understand?

Much like how things progressed better in heavier-than-air flight once engineers realized that flapping wings weren't absolutely needed, and low-weight engines turning propellers, with fixed wings, worked better for flying than what nature uses...?

The reasons to use ReLUs are sparsity and improved gradient flow. ReLUs encourage sparsity because when the input to the ReLU is less than 0 as the activation becomes 0. This means some fraction of activations in a given layer will be omitted which can encourage better representations. They also have improved gradient flow because the gradients are zero or constant and thus don't suffer from vanishing/exploding gradients.

In deep learning, I would _generally_ not look towards biology for the reasons behind why things are done as this is usually an after-the-fact explanation. When in doubt, blame the gradients.

LeakyReLUs work as well or better than ReLUs, so it can't be because of sparsity.

LeakyReLUs often have very small slopes on the negative side; this can help solve the problem of no gradients getting through a layer because the activations were all 0. But I would say it still has a sparsity effect on a trained network because of how small the slope usually is compared to one.

That is not sparsity. In machine learning there is a very strong distinction between values that are exactly 0, and values that are close to zero (see: difference between L1 and L2 regularization).

L1 regularization does not lead to values that are _exactly_ 0 either.

Yes, it does. You might have been thrown off by the fact that L1 regularization is not L0 regularization, i.e. it doesn't explicitly limit the number of nonzero coefficients. Still, the linearity of L1 constraint boundaries creates spikes in directions with zero components, thus forcing constrained solutions to occur where many variables are driven to exactly zero. See here:

https://en.wikipedia.org/wiki/Lasso_(statistics)#Geometric_i...

If we have a parameter x, and some cost function J(x), then with L1 regularization the cost function would be J(x) + beta * abs(x).

The derivative of that loss with respect to x would be J'(x) + beta * sgn(x). So using some variant of SGD (which is what basically all neural network training does these days) we would essentially update x as: x = x - alpha * (J'(x) + beta). (The specifics depend on the algorithm, but it doesn't change the result).

So for x to end up as _exactly_ 0, we have to be extremely lucky, which in practice I have never observed. Using L1 regularization definitely leads to small weights, but not to ones that are _exactly_ 0.

Your conclusion is theoretically false.

You can prove that L1 regularization is equivalent to taking the optimal unregularized parameters, setting to parameters below a threshold to 0 (the threshold depends on the regularization parameter), and penalizing the other parameters.

Yes, but how do you actually optimize that loss in practice? I'm not saying that a perfect solution with an L1 penalty wouldn't have weights exactly equal to 0. I'm saying that with the optimization techniques that are commonly used, you don't end up with exact zeros.

You're not making sense. If the loss function is convex, adding L1 regularization is still convex. So iterative methods for convex problems (which include SVMs, linear regression, and logistic regression) will find the global optimum.

1) Neural Network Loss functions are not convex. But that isn't the issue here.

2) When you use actual numerical optimization techniques with floating point arithmetic, you don't find an exact minimum (global or local). And you don't get exact zeros.

Have you tried this on a real problem? I wouldn't consider MNIST a real problem, but even there you will not get _exact_ zeros. Try it.

If you rolled your own naive numerical approximation of L1 regularization, you might not have gotten exact zeros. If you use e.g. LARS or cyclical coordinate descent for the L1-regularized parameter cohort, as suited to the problem, you will get exact zeros, as prescribed by the mathematics of L1.

I've never seen anyone optimize a neural network using LARS or cyclical coordinate descent. I thought that's what this entire discussion was about - not arbitrary optimization theory.

The sparsity encouraged by ReLUs is one reason they are used but not to the exclusion of all other activations. ReLU variants can indeed outperform standard ReLU[0] but sometimes sparsity is a desired property. For example, in generative models.

[0] https://arxiv.org/abs/1505.00853

RELU is decidedly the more "biologically accurate" (or at least biologically inspired) compared to the sigmoid function. The idea put forth to originally justify RELUs was that neurons in the brain only have action potentials or lack thereof: the lowest level of activity a neuron can have is zero, and then you can ramp up frequency of action potentials from there, as opposed to the sigmoid's positive and negative symmetric curves.

That said, a lot of the history of neural networks has been brief moments of biological inspiration followed by hacking and engineering that drifts further away from the biology the better it gets. The biggest example is backpropagation; despite how essential it's been to artificial neural networks, it really doesn't exist in the brain, at least not as simply as it does in code. For now, we're all still exploring, some looking towards biology, some towards abstract principles, and it remains to be seen if one provides consistently better results.

> The biggest example is backpropagation; despite how essential it's been to artificial neural networks, it really doesn't exist in the brain, at least not as simply as it does in code.

Do you have any links to papers or such explaining "at least not as simply as it does in code"?

EDIT: nvm, I followed the links in a wikipedia article on RELU to relevant PDFs...

For a while now this is one area I have been questioning - that we do use backprop, and maybe there is something to be learned from nature that might (?) simplify how a NN is trained (then again, nature might be doing it in such a way that is more complex than can be engineered or practical)...

> For now, we're all still exploring, some looking towards biology, some towards abstract principles, and it remains to be seen if one provides consistently better results.

It might end up being a combination; at least, that seems the direction so far to a point.

I want to thank you for your comments, though. I'm still learning this stuff (I'm working thru the Udacity Self-Driving Car Engineer Nanodegree), and you've given me some stuff to think about and explore further.

Honestly, the current state of neural backpropagation research is still in flux and papers will be pretty technical. My background's in neuroscience, and the rough summary of what we know so far is: action potentials can travel backwards down a neuron, and they probably do so in vivo, but we don't know if this actually does anything. There's no mechanism we understand for passing information backwards through a synapse. In fact, the only reason people suspect this might be important, and not just an accidental effect, is because backpropagation has been important in developing artificial neural networks.

Getting inspiration from nature is often useful (convolutional neural networks are loosely inspired by the way information is processed down the optic nerve) but for the specifics of neuron function, the brain is almost certainly doing things that are more complex than is practical to simulate. The brain contains hundreds of billions of neurons, and each one is orders of magnitude more complex than a node in an ANN, interacting with local and distant neurons through all kinds of separate but interlocking electrical, chemical, and long-term genetic actions.

Regarding "brain-like" propagation strategies:

Is there any NN model which tracks a "supply" each neuron has of signalling "molecules", such that a given neuron won't be able to communicate a message of class X if it's out of X-amine, unless/until it receives some from a neighbour?

Or, is there any NN model that allows each neuron to send broadcast messages, ala extracellular ionic neurotransmission, which decay with "physical distance" (requiring each node on the neural graph to have a physical position) and which remain active until "sucked up" by something?

I've always thought these two things—neurons needing to "spend" chemicals for neurotransmission, and neurons being able to observe the local-in-physical-space "loudness" of certain broadcast signals—seemed to have high potential for modelling agentive systems generally, since they seem to replicate other successful systems (markets; pheromonal communication), but I've never heard of either concept being studied in an ML context.

The most mainstream of the "biologically-inspired" Artificial Neural Network variants is the Spiking Neural Network model, which IBM has been implementing in their TrueNorth chip architecture[0], and APT implements in their SpiNNaker[1] design.

That said, this is only from the perspective of constructing ANN's for computational purposes. There are, of course, many detailed models of various processes in the brain, but this is generally under the heading of Computational Neuroscience[2] (though the field boundaries are blurry). The main software for simulating biologically accurate neural networks is "Neuron"[3], which is used in the Blue Brain Project[4].

[0] https://en.wikipedia.org/wiki/TrueNorth

[1] https://en.wikipedia.org/wiki/SpiNNaker

[2] https://en.wikipedia.org/wiki/Computational_neuroscience

[3] https://en.wikipedia.org/wiki/Neuron_(software)

[4] https://en.wikipedia.org/wiki/Blue_Brain_Project

Would an action potential traveling backwards be able to block the neuron from properly firing, briefly? In other words, could this effectively act as a short term inhibition?

Thank you for this explanation - I would be curious about any papers, no matter how technical. I've never let something like that stop me before...

Nature uses something like STDP (spike timing dependent plasticity), where the change in a synapse strength is proportional to the sign and delta time of the last spike at each end of the synapse. You can approximate something like backprop by using symmetric STDP (discarding the sign).

"Towards a Biologically Plausible Backprop" from Benjamin Scellier and Yoshua Bengio (2016) would be a recent paper on the topic.

> "Towards a Biologically Plausible Backprop" from Benjamin Scellier and Yoshua Bengio (2016) would be a recent paper on the topic.

Thank you for this link; I've found a copy of this paper - and some others - and will be reading them with interest!

What's interesting is that Jeff Dean noted that ReLU was one of the big things why deep learning took off. Wikipedia summarizes tha advantages, but my guess is getting rid of the vanishing gradients problem:

https://en.m.wikipedia.org/wiki/Rectifier_%28neural_networks...

> Wikipedia summarizes tha advantages, but my guess is getting rid of the vanishing gradients problem

Hmm - didn't think about that aspect, but that would be a big plus for backprop as I understand it! Thanks for the insight, and the link.

https://smerity.com/articles/2016/google_nmt_arch.html section "V6: Residuals are the new hotness". You can think of each layer (or I guess a pair of layers) as trying to correct errors in the previous layer. If the previous layer is close enough, the next layer can output 0. This is more stable than a straight linear activation, where one layer might move up a bit as the next one moves down to compensate! ReLU lets the earlier layer fine-tune itself without the effect being undone by later layers.

Interesting! Thanks for the link!

Gradient flow due to chain rule is the reason. Also that last part is lecun a analogy ;

Reply on news.ycombinator.com