A new trick for calculating Jacobian vector products | Hacker News Reader

A new trick for calculating Jacobian vector products

j-towns.github.io

76 pointsrosstex14 comments

I love the discussion thread on GitHub: https://github.com/HIPS/autograd/pull/175#issuecomment-30652...

As someone just starting to study machine learning it's inspiring and exciting seeing this kind of thing.

Can somebody explain why this is interesting computationally-wise? It looks like standard derivation to me.

Normally reverse mode automatic differentiation is used and is called backpropagation since everything old has to have a new and shiny name in machine learning a/k/a continuous optimization ;)

I concur. When I first started taking online coursework in machine learning, I was in almost every unit asking myself, didn't I cover this already in my non-linear optimization class at Uni.?

The main difference is that a lot of practitioners using ML have very little advanced mathematical background. Let's be honest, you can't understand much of optimization without at least multivariable calculus and linear algebra.

I think this is a very bad thing, but basically what happens when things become "commoditized".

I feel it's the same with web developers who have no formal background in Computer Science. All these frameworks reinventing "paradigms" that have been studied since the 60's.

I'm not sure what your point is. People use tools without understanding the underlying principles all the time, frequently in ways that are more (tangibly) practical and interesting than a person who does would use them.

Presumably his point is that you can be more effective at something if you understand it at a level one level below the one at which you are working.

I'd say that depends entirely on context. It also has diminishing value each "level" one goes. I mean, understanding quantum degeneracy pressure doesn't make physicists more effective at being solid.

I think his point is about reinventing the wheel due to insufficient theoretical background.

ML researchers have the math background.

I have had the same thoughts too. What also had me a bit puzzled in the ML community is the lack of discussion on local optimality, and constraint/Lagrangian loss functions. But there is probably a subtlety that I'm missing.

The engineers (aerospace, etc) also prefer the term "adjoint mode"

It's the first open source implementation compatible with the modern ecosystem. Software development isn't about novel math, it's about novel applications and deployments

I've always been curious why none of the silicon companies have attempted to provide hardware (or at least ISA extensions) supported AD, backprop, etc.

Reply on news.ycombinator.com