Comment by jerf | Hacker News Reader

Comment by jerf | Hacker News Reader

Parent Full thread

Here is the public leaderboard for the now-completed contest, where they are in first place out of 2,257: https://www.kaggle.com/c/restaurant-revenue-prediction/leade...

Here is the private leaderboard, which I believe represents the actual score of the contest, where they are in place 1,962: https://www.kaggle.com/c/restaurant-revenue-prediction/leade...

That's, ah, not in the prize range, to put it lightly.

There's no problem with the contests here. It's just a particularly vivid, and entertaining to humans, demonstration of the well-known problem with overfitting.

If I am reading this correctly, the green numbers to the left of the teams in the private leaderboard represent that team's position relative to the public leaderboard. Note they're quite large... the 9th place team in the final scores jumped 1,058 positions! That's nearly half the field. If this is typical, and I have no idea (reply & let me know!), the public leaderboards are basically a joke.

It depends on the competition. For this competition it was to be expected as: There was little training data and there were a lot of outliers (the problem was predicting restaurant profits). Luck played a huge role here. The 9th place team likely had a mediocre highly-variant model that got a very lucky leaderboard split.

This forum post discusses competition variance and poses a metric to quantify "leaderboard shake-up": https://www.kaggle.com/c/liberty-mutual-fire-peril/forums/t/...

The Public Leaderboards are very helpful though! When you have setup a solid local cross-validation pipeline, and the public leaderboard agrees with your local evaluation, then you can try a lot more algorithms and parameters, without using any submission. Especially when working in teams this is important as you may have only 1 submission every 2 days.

Also, the more advanced Kagglers can use leaderboard feedback to increase model accuracy: Cluster the data sets with objective measures. Apply a modifier (restaurants from this region get 0.95 x previous prediction) and look at the result. If the split between public and private is random, and your clustering is objective, then an improvement on public leaderboard should reflect in private leaderboard.

Still, this also confirms the obvious fact that the selection of the winning team based on that hidden, private test also gives no guarentee that it is not accidently overfitted, especially with tons of submissions. The only reliable option to solve it is to ask participants for self-contained programs and run it inside a CV loop on the organisers' hardware, but this seems too cumbersome to be implemented in reality; I think only TunedIT tries it, but without luck :L

So, why have 2 leaderboards? Or the private one has limited visibility by contestant?

(there seems to be simple ways for kaggle to solve this: put a delay on the score generation, limit number of submissions, limit precision of scores on the public leaderboard)

I'm guessing that the private one isn't visible to contestants or the public until the contest is over (hence "private").

The published dataset gives contestants an idea of how good they are, but prevents gaming the system by overfitting, because the contest is actually scored with the second unknown dataset.

Even with those other methods in place, the leaderboard would still favor methods that overfit. That is why the final score is determined on a dataset (the validation set) that is not used until the model is locked down. The public leaderboard is based on the test set.

A more in depth explanation on the use of three datasets for model building can be found here: http://stats.stackexchange.com/questions/19048/what-is-the-d...

The private one is revealed after the competition is over. Only the private leaderboard counts. To get an idea of how well your model performs, you get scores for, let's say 30% of the holdout data, on the public leaderboard. You can also use local cross-validation to gauge model performance.

There is a limit on the number of submissions. Usually 5 or less a day. Also precision of scores is visibly limited, but for final ranking all decimals count.

Reply on news.ycombinator.com