Hacker News Reader

Comment by wyc | Hacker News Reader

"Baidu does not show up in this year’s rankings. The company made more submissions than were permitted and ultimately apologized and fired the team leader who directed juniors to make the unacceptable submissions."

View on HN

see http://dswalter.github.io/blog/machine-learnings-first-cheat... for some background on this.

Pretty good read, provides all the context needed and raises some important points about benchmarks. Thanks for sharing.

now discussed at https://news.ycombinator.com/item?id=10730020

Might be worth noting that it is very likely that Google, Facebook and Baidu did not even participate in this ImageNet. It is widely believed in the vision community that this dataset is at the end of its run, i.e. any subsequent improvements in performance are not due to scientific breakthroughs but through hyperparameter optimization and ensembling.

The article and the title specifically mentions Google.

Google participated with ReCeption / Inception v3 model, see the results: http://image-net.org/challenges/LSVRC/2015/results

In classification task, MSRA and ReCeption show very similar performance: 0.03567 vs 0.03581 top(5) error. The gap is much more drastic on the localization task: 0.090178 vs 0.195792

The residual learning presented by Microsoft Research seems to be a breakthrough, based on the early evidence. But yes, ImageNet needs to be updated to stay relevant.

Surely that's not true until someone achieves human-level performance with their entry.

Human level performance was surpassed earlier this year:

http://arxiv.org/abs/1502.01852

I won't be surprised when human level performance is surpassed, but isn't that paper based on work that has the advantage of knowing the full test dataset in advance? You can say "yes but they only trained on the training data" but that doesn't rule out tweaking across several experiments and measuring them against the known test data, then cherry picking the experiment that was best overfit to that data, right? I'm not saying there are shenanigans here, just wondering how you know there are not.

The test labels are not known to them. The test labels are held by a different team that tells them their test error rate. Furthermore, multiple, separate teams have surpassed human performance.

FWIW, "human performance" is not as meaningful as most people make it out to be, because this is such a narrow task.

The paper mentions it's unfair to humans to have to tell say a coucal from an indigo bunting (both blue coloured birds). I've never heard of either so I guess I would have got that wrong but it's not the fault of my visual system.

But you only have to guess right once in 5 tries!

I replied to abrichr's link. Did you look at it at all?

They used a dataset from 2012. Are you saying that the test labels from a 2012 dataset have been kept secret until now? Really?

I am familiar with the ImageNet competition. Let me know if you find the ImageNet test labels. (Hint: they are not released, since they are re-used for competitions).

Interesting, did not know that. Do you think that proves this team was not able to get and exploit some inside knowledge? Judging from recent competitions, ethics does not seem to be a strong point in some cultures.

Did they train also the human on the training data ?

Google entered with ReCeption and finished just after the Microsoft team in the "Ordered by classification error" section (which is the usual "headline" number).

Baidu was banned. I don't think Facebook has ever done ImageNet?

But I agree that most of the interesting work will happen outside ImageNet now that human performance has been comprehensibly surpassed.

The only exception is non-Deep Learning based systems, where some people remain convinced that alternative approaches can match DL systems and have other advantages.

I bet there are breakthroughs to be had if they would compete on training time and computing resources, rather than just accuracy.

Reply on news.ycombinator.com