Hacker News Reader

Comment by alexmolas | Hacker News Reader

The results of the experiment seem counterintuitive just because the used learning rates are huge (up to 10 or even 100). These are not the lr you would use in a normal setting. If you look at the region of small lr it seems all of them converge.

So I would say the experiment is interesting, but not representative of real world deep learning.

In the experiment, you have a function of 272 variables with a lot of minima and maxima, and at each gradient descent step you take huge steps (due to big lr). So my intuition is that convergence is more a matter of luck rather than hyperparameters.

View on HN

If convergence were a matter of luck, it would look completely different, like white noise, but it clearly has well-defined structure.

The reason for high learning rate is that they used full batched training (see the first cell in https://colab.research.google.com/github/Sohl-Dickstein/frac...), and when batch sizes are large, learning rates typically can be large as well. Plus as others said it's more of a toy problem, it would be hard to get such detail on anything non-toy.

I think the article is very honest about this just being a fun exploration. They even show how you can get similar patterns with newton's algorithm which is a more "classical" take

Yes, the author is clear in this regard. But I've seen people interpreting this paper as "training deep networks is chaotic", and I don't think that's the case. I interpret it more as "if you're not careful with your learning rate your training will be chaotic"

Yes. I immediately had visions of a misconfigured PID controller or similar chaotic emergence encountered in control theory. I was surprised and delighted though, that this type of chaos has so much fractal beauty.

Note that not all fractals are chaotic.

Inter dimensionality is the defining feature of fractals.

A multilayer ANN will have compression that may be fractal like. But in theory a feed forward network could be single layer but will a lot more neurons.

I have the feeling this result was due to representation but there are some features that look like riddled basins.

Riddled basins do arise in SNN or spikey neutral networks that have continuous output bs the binary output of ANNs.

But as all PAC learning is just compression that may be the cause also.

I'll be digging into this on the weekend though.

With all do respect, why are you commenting here rather than replying to the person giving your comment context?

There was a nice paper about five years ago that also found that the best hyperparams were in the bounty of divergence, even for 'real' imagenet models.

What paper was that?

Here you go: https://arxiv.org/abs/1811.03600

Yes, but as he also says, the best hyperparameters are near the boundary. You want to be a bit greedy here.

The way the author defines best hyperparameters is also a bit weird. To assign a "score" to each pair of HP the author sums all the losses during training (ie score = ∑ᵀₜ₌₀ lₜ) instead of just taking the value of the loss at the last epoch (score = l_T). This makes the converging solutions with high learning rates appear better since they have lower losses since the beginning (so the sum is smaller), but it doesn't mean that the final value of the loss is better than the ones with smaller learning rates.

Reply on news.ycombinator.com