Hacker News Reader

Comment by TuringTest | Hacker News Reader

Long story short:

- The LSVRC "visual recognition" competition has a rule in place, that limits to twice per week how often each contestant can run their entries to the contest against the ImageNet dataset.

- The Baidu team run their tests much more often, claiming that they understood the limit was placed on a person basis and not per team.

- This rule is in place because more frequent submissions somehow distort the quality of the test, by switching its emphasis to overlearning (adapting too much to the specific data included in the test dataset) instead of true algorithmic advances, which can influence the whole machine learning discipline given this contest's high profile. ( * )

- As a result, the whole Baidu company has been banned from the competition for a year.

( * ) ("The Baidu team’s oversubmissions tilted the balance of forward progress on the LSVRC from algorithmic advances to hyperparameter optimization.")

View on HN

Do they test against the whole dataset? Wouldn't that be fixed by having a test dataset and a separate, bigger dataset that would be used for the final grading (like on Kaggle)?

They have a separate test set. But by submitting a lot of entries with slightly different parameters, you can optimize your parameters based on the test score.

Some (all ? ) Kaggle competition also have a daily submission limit to avoid this kind of cheating.

I think you misunderstand. Kaggle has a training set (known to all participants), a "public leaderboard" test set (secret) and a "private leaderboard" test set (also secret). You can get your model's score on the "public" test set a couple of times per day. Your score on the private test set is only revealed once, after the competition has ended.

People can, and do, overfit their model to the public test set, but doing so does not improve their score on the private test set, so cheating is prevented even without the submission limit.

The submission limit helps ensure the leaderboard generated from the public test set stays close to the leaderboard generated by the private test set while the competition is running, so that you can get an idea about your standing. But participants know better than taking the public leaderboard too seriously.

I stand corrected :) It seems like ILSVRC only has 2 datasets though, I wonder why they don't use a Kaggle-like approach ? My guess would be because they want people to be able to know their test score in order to put them in their papers.

The grading test set is secret.

This is what the Baidu team were probing.

I should have read the comments, I just asked the same thing. I always wondered how that worked, though - you don't submit an algorithm, do you? Just a spreadsheet with your predictions, in which case how can they evaluate your algorithm on another test set?

You get to see the feature vectors of the test set, or at least submit your algorithm to execute upon them. If you can see the features of the test set but not the labels or target variable for each feature vector, then you can't use it to help you (e.g. the problem of modeling the distribution over outcomes for a given feature vector (whose target outcome you don't know) is the problem you're solving).

But if you can repeatedly run different models against the test set and you do get to see a score, you could do something like random parameter searching or other optimization ideas to tune your algorithm to be highly overfitted to the test set.

Another way to avoid this would be to develop several test sets that are roughly "equivalent" in terms of the distributional properties, and then randomly change the test set periodically, or change the test set right after the final submission deadline, to discourage people from pursuing overfitting.

That makes sense. I was thinking of the multiple test set approach, but rather than forcing people to re-submit on a new test set near the end (if that's what you meant), the organisers could just return the score based on a random fraction of the test set. 'Cheaters' would then overfit to just half of the actual test set and this would be apparent when the final scores (on the other half) are revealed. I suspect this is what Kaggle do, as they don't make you re-submit at the end (AFAIK).

Publish the final test set right before the deadline.

Alternatively, require the participants to provide an API and call them, instead of them submitting things.

Reply on news.ycombinator.com