Hacker News Reader

Comment by alquemist | Hacker News Reader

The title is misleading. The core technique still uses 60,000 images from MNIST, but 'distills' them into 10 images that contain the information from the original 60,000. The 10 'distilled' images look nothing like digits. Learning a complex model from 10 (later reduced to 2) 'distilled' number arrays is an interesting research idea, but it has little to do with reducing the size of the input dataset. Arguably the heavy lifting part of the learning process moved from training the model to generating the distilled dataset. There is also some unconvincing discussion around synthetic datasets, though it remains fully unclear how these synthetic datasets have anything to do with real world scenarios.

> In a previous paper, MIT researchers had introduced a technique to “distill” giant data sets into tiny ones, and as a proof of concept, they had compressed MNIST down to only 10 images.

View on HN

It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere in this.

"Upgrade packs" might be already possible using transformer adapters, i.e. tiny networks trained on customized data plugged into a large fixed pretrained transformer, providing whatever custom functionality you require.

I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.

For aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.

It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns.

The total entropy in 10 images (even carefully engineered ones) is very low in comparison to the full data-set.

So this is more like data storage/some sort of extreme compression than anything? Would it be accurate to interpret this as basically saving the trained neural network into 10 images and reading it back to retrain a model? Or, what really does this do/accomplish?

mostly it seems that it tells you something sort of weird about how neural networks train. it’s not obvious that this should work, and that it can be made to work is interesting.

Agreed, the idea is interesting and worthy of further exploration. For starters, does it scales beyond MNIST or 'tiny synthetic datasets'?

The approach appears similar to model distillation, performed on the input data instead of on the weights. Model distillation scales well in classification but less well in generation.

To me, this seems similar to training a neural network on 10,000 images and then publishing the weights themselves as a distilled "image" (of course, it's not the weights themselves but a "distillation"). I feel like you could get something similar by taking the trained network and then applying deep-dream techniques for each classification label on a blank canvas.

It's interesting for localized learning... You have some local data that you want the model to incorporate, but don't want it to forget the main dataset. So you can create one of these distilled datasets and include it in a batch with the local data when you updated the model.

At least I'm glad that the press article included the actual images. Oftentimes we have to track the original paper to see that kind of thing.

Ok, we've replaced the title with the first part of the subtitle.

Is this process net faster, or requiring of less memory?

Reply on news.ycombinator.com