Hacker News Reader

Comment by throwawayiionqz | Hacker News Reader

I am wondering what is the state of things for fixing gradients are kinks, or if there is no hope for general automatic differentiation libraries.

    import jax.numpy as np
    from jax import grad, jit, vmap
    from jax import random
    
    def relu(x):
      return np.where(x>0, x, np.zeros_like(x))
    
    def identity(x):
      return relu(x) - relu(-x)
    
    derivative_identity = grad(identity)
    derivative_identity(0.0)

returns 1.0 or 0.0? It currently returns 0.0. (Edit: typo in sign)

View on HN

For many kinks, it doesn't in practice seem to matter that much. Most applications are computing large stochastic derivatives which smooth out kinks through averaging.

Critical kinks are those that affect the geometry of the gradient in systematic ways. For instance, a model with a mixture of discrete and continuous parameters. These are serious blockers and require more complex methods to solve such as Rao-Blackwellization (marginalizing out the discrete parameters). Generally this appears as model bias or substantially increased, often fatal variance in loss curves.

See https://en.m.wikipedia.org/wiki/Subderivative

The derivative of x->x (identity) should be 1 regardless of your definition of derivative.

Subgradients are only applicable when summing convex functions. Here relu(x) - relu(-x) is a sum of a convex function and a concave function.

0.0>0 is false.

Reply on news.ycombinator.com