Hacker News Reader

I think you misunderstand. Especially

> this model contains a smaller model, that has similar performance to the trained large model, without training

The point is the opposite. There is a small net X within big net Y, such that training only X gives the same performance as training all of Y.

View on HN

What you are stating is the original Lottery Ticket Hypothesis. What they prove in this paper is the stronger version, empirically noticed here https://arxiv.org/abs/1905.01067 and referred to as "supermasks". To quote from the paper posted here: "within a sufficiently overparameterized neural network with random weights (e.g. at initialization), there exists a subnetwork that achieves competitive accuracy".

Edit: See also https://arxiv.org/abs/1911.13299

Seems like a "Library of Babel" type of thing. I'd have to read the full paper for how they find the subnets, but their mere existence is not so surprising. There's a huge sea of possible subnetworks. Basically SGD is replaced by whatever procedure you use to traverse the space of parameter subsets. Definitely interesting direction.

Reply on news.ycombinator.com