Hacker News Reader

Comment by lawrenceyan | Hacker News Reader

Local minima are basically equivalent to global minimum in the high dimensionalities achieved in deep learning. It's not even just a conjecture by the way. This has been mathematically proven.

View on HN

Is there a name for this theorem? I'd be interested in reading more about it.

Maybe GP is talking about the hypothesis that in high-dimensional space you will probably almost always find /some/ direction which decreases the error function. IIRC this hypothesis is well supported, but not yet proven. So the real problem may not be local minima but saddle points that just /appear/ to be minima due to slow optimisation progress. For more details, check out https://arxiv.org/abs/1406.2572

No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis [1]. Authored by Rong Ge, an absolute prodigy in the field if I might say. I'm sure we'll be seeing a great deal more from him in solidifying the theory behind the current day trial and error sort of deep learning we have today.

[1] https://arxiv.org/abs/1704.00708

He also did “On the optimization landscape of tensor decompositions” with Tengyu Ma which I thought was really cool.

Definitely, I second a read on this paper in addition to the first.

If I understand the Wikipedia page for SGD correctly, it’s the Robbins Siegmund theorem.

That theorem only implies global convergence for convex or pseudoconvex functions. For a generic function it only guarantees convergence to a local minimum.

Could you link to your source?

I recall seeing some research on this topic but I got the impression (perhaps wrongly) it has to make so many simplifying assumptions to get a result that it’s not clear how much carries over to real systems where not all those assumptions hold.

Reply on news.ycombinator.com