Hacker News Reader

Maybe GP is talking about the hypothesis that in high-dimensional space you will probably almost always find /some/ direction which decreases the error function. IIRC this hypothesis is well supported, but not yet proven. So the real problem may not be local minima but saddle points that just /appear/ to be minima due to slow optimisation progress. For more details, check out https://arxiv.org/abs/1406.2572

No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis [1]. Authored by Rong Ge, an absolute prodigy in the field if I might say. I'm sure we'll be seeing a great deal more from him in solidifying the theory behind the current day trial and error sort of deep learning we have today.

[1] https://arxiv.org/abs/1704.00708

He also did “On the optimization landscape of tensor decompositions” with Tengyu Ma which I thought was really cool.

Definitely, I second a read on this paper in addition to the first.

If I understand the Wikipedia page for SGD correctly, it’s the Robbins Siegmund theorem.

That theorem only implies global convergence for convex or pseudoconvex functions. For a generic function it only guarantees convergence to a local minimum.