Findings from the Imagenet and CIFAR10 competitions | Hacker News Reader

Findings from the Imagenet and CIFAR10 competitions

fast.ai

85 pointsjph0017 comments

> I worry when I talk to my friends at Google, OpenAI, and other well-funded institutions that their easy access to massive resources is stifling their creativity. Why do things smart when you can just throw more resources at them?

I'm not convinced that this is a real problem. The big tech companies have way more compute than anyone else, so they should do the large-scale experiments that no one else is able to do. It's true that you don't have to be particularly creative to come up with a lot of these experiments, but nevertheless they should be done and there is a lot of scientific value there.

I absolutely agree that more attention needs to be paid to smaller institutions since they're just as likely as anyone to come up with the next great idea. Double blind conference submissions help with this to an extent, but they're not a panacea. If you see an anonymous paper that has performed a thousand Imagenet experiments, is there really any doubt where it came from? And similarly, if they didn't perform a bunch of Imagenet experiments, is there really any doubt that it didn't come from one of the big players? So now you can mask a bias against small institutions as a subtler "you didn't perform enough Imagenet experiments" excuse. (Incidentally, this was the main reason that the paper by Smith & Topin was rejected from ICLR [1].)

EDIT: Full disclosure, I'm currently working on a set of not-so-creative (but still IMO important) experiments that use a large amount of compute at Google. So I have some bias here. :-)

[1]: https://openreview.net/forum?id=H1A5ztj3b

Jeremy from fast.ai here. Let me know if you have any questions about the methods we used, or anything else relevant to this project.

Hi Jermey... Great article. I have a labeled product image dataset (thumbnails, and various sizes) that I'd like to use for the model training. Can you recommend any lightweight methods for commodity hardware?

You can use the approach we used to win the CIFAR 10 training cost section of the competition. If you use fastai/pytorch, then it's ~5 lines of code. Check out lesson 1 of http://course.fast.ai for the basic approach, but when calling `fit()`, add the param `use_clr_beta=(20,20,0.95,0.85)` which will enable 1cycle, and should allow of super convergence. Then train with SGD with a really high learning rate (somewhere from 1-3, generally).

Excellent. I will check it out.

I do have a question, but first I must say that as I was reading your post I found myself nodding in agreement again and again and again. Yes, the Silicon-Valley-centric AI research community (including UK-based DeepMind and its surrounding satellites) tends to overlook or even ignore important research from elsewhere. Yes, "algorithmic creativity" and focused tinkering often prove more important than computational resources. Yes, innovation in deep learning does not require big data nor massive hardware, but "engineers are drawn to using the biggest datasets they can get, on the biggest machines they can access, like moths flitting around a bright light" (so true!). And yes, "genuine advances [have] consistently come from doings things differently, not doing things bigger." Ah-men!

With that out of the way, here's my question:

Have you guys tried or managed to achieve learning super-convergence with attention or residual attention models?

No I haven't tried yet. One of our students has started experimenting with language models (AWD LSTM) and getting encouraging results (about 3x speed improvement), but not with attention.

I really hope that these results will help encourage more people to both see where and how super-convergence can be achieved. I suspect that we're only scratching the surface of what's possible.

Thank you for the prompt answer -- and for your intellectual generosity.

I too hope these results encourage more people to see where/how super convergence can be achieved.

FWIW, trying to achieve it with fully attentional models is on my "R&D things to try" list at work.

Does the solution use pre trained Resnet models?

No you can't use pre-trained models in this competition.

Thanks for the prompt response. Do you use data augmentation? Sorry for all the questions which could maybe be answered by looking at the code, but I am just getting started with pytorch etc and I am trying to understand the exact input format.

Yes data augmentation is critical

How important was momentum cycle (The decrease then increase)?

Unrelated but it would be great if you could answer: In general, how important is momentum tuning and what are the heuristics for the same?

How important was momentum cycle (The decrease then increase)?

Unrelated but it would be great if you could answer: In general, how important is momentum tuning and what are the heuristics for the same?

Based on Smith's paper and our experiments, it seems pretty important. But there's still much research to be done to understand momentum tuning and related concepts.

Is the code used for the competition available?

Yes it's linked from the competition web site. It'll be integrated into the fastai library in the coming weeks.

Reply on news.ycombinator.com