Hacker News Reader

Stuff got stuck in local minima for years before we learned about stuff like momentum and dropout and dropped a ton of GPU power on it.

View on HN

When I was implementing a neural network for a university assignment (2 years ago so my memory might fail me), we had to run our algorithm multiple time with different starting positions, then take the minimum of those local minima.

I'm not sure what momentum and dropout are, but I agree with Eleizer, without these things (which I didn't use) local minima are a problem.

Dropout is where you randomly remove neurons from your network during training, which prevents them from depending too much on specific neurons (making the output more generalizable). It was developed in 2014 so it would have been brand new tech back when you were in your class.

I think you're misinterpreting the parent who is saying that local minima are not a problem in high dimensions because there is always a dimension to move in that reduces the loss (unlike in lower dimensions where you can get stuck in a point across all dimensions that cannot be locally improved upon)

I still don't understand what the parent is talking about then. Could you please restate the explanation using math notation/terminology?

Reply on news.ycombinator.com