Hacker News Reader

Comment by amelius | Hacker News Reader

Hmm, I've just scanned through this, but it seems this gets the concept of stochastic gradient descent (SGD) completely wrong.

The nice part of SGD is that you can backpropagate even functions that are not differentiable.

This is totally missed here.

View on HN

No, stochastic gradient descent (SGD) needs a differentiable loss function (differentiable with respect to the trainable parameters). The stochastic bit is that you want to optimize the loss for the whole dataset but what you actually do is to do each gradient descent step on a small mini-batch of the dataset.

There might be extensions where you have some non-differentiable part where you just assume some gradient in back-propagation. Like Gumble-Softmax (https://arxiv.org/abs/1611.01144). Or similar is the reparametrization trick in VAEs (https://arxiv.org/abs/1606.05908). But that are special cases.

Maybe you are thinking about reinforcement learning?

How do you compute the gradient of a non differentiable function? I'm not an expert, but that contradicts anything I've learned about gradient descent.

Pseudo gradients are a thing. You can pretend its a continuous function and get gradients that push it in the right direction.

As others have pointed out, SGD requires differentiability. The next best match I can think of is that you're actually thinking of subgradient methods, which mostly see use in convex optimization problems.

Can you explain? I thought the entire point of back-propagation was to differentiate and calculate the weights that are contributing more to the error and thus will be "changed" more when you do GD.

maybe you're thinking about simulated annealing?

Reply on news.ycombinator.com