Comment by SpaceManNabs | Hacker News Reader

Comment by SpaceManNabs | Hacker News Reader

Parent Full thread

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten.

Confused by this statement. Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books.

> In their new proof, the pair show that overparameterization is necessary for a network to be robust.

What is important to note here is that many of papers this paper cites prove or show this result in certain network architectures. This paper adds universality.

> The proof is very elementary — no heavy math, and it says something very general

The most elementary part was clever use of Hoeffding's inequality. Some people are really fast readers haha.

I don't even know how you pick up the fact that isoperimetry holds in manifold settings with positive curvature while also playing with all those norms and inequalities. A few years ago I mentioned on here all the maths that I knew or wanted to know to read more papers, and others critiqued that the list was too long. Well, this is why!

> Double descent with overparameterization is exhibited in "classical settings" too and mentioned in older books.

I’m curious for references or citations to this. When I was going over double descent I tried to find citations like this (just in a couple places like ML/stats textbooks).

There are a handful of papers in the 90s that show this, but it wasn't recognized for what it is. Double descent is REALLY crazy to me, coming from a classical background.

Over-parameterization for regularization is really old. The pseudoinverse min-norm solution for under-determined linear systems even has that flavor.

Sure, but that's identification approaches in econometrics and matrix analysis contexts. Using that for neural networks is new-ish in the zeitgeist, which did not exist in the 1990s as it does today.

Here's one that lists some older references: https://arxiv.org/abs/2004.04328

Here's a very good visual explanation of double descent:

https://mlu-explain.github.io/double-descent/

> ... "the model perfectly memorizes ... the training data."

Can you call this learning?

Daniela Witten has a nice twitter thread on the topic of double descent in classical settings: https://twitter.com/daniela_witten/status/129229310485515878...

> Right now, we are routinely creating neural networks that have a number of parameters more than the number of training samples. This says that the books have to be rewritten.

I also believe that this statement is weird. I have a very shallow knowledge of ML, but I can imagine that in a convolutional neural network a training sample interacts with lots of parameters. This 'one training sample <-> one parameter' correspondence seems wrong to me.

One CNN parameter would interact with the total number of training samples, but that doesn’t tell you about the ratio of total training samples to total CNN parameters.

Reply on news.ycombinator.com