Comment by harry8 | Hacker News Reader

Comment by harry8 | Hacker News Reader

Parent Full thread

tl;dr

Competive ML model grading with a common training set using unseen data.

Cheat was to scrape the data that would then be used as unseen by the organisers. Unseen is now seen for this model. Then, instead of training the model with the "unseen" data, which would have been cheating and an advantage it apparently wasn't enough of an advantage so they hard code 10% of the cases to boost metrics and win.

Having more data to train your model is google & facebricks competitive advantage. Their attempts to use that advantage for something actually useful to society rather than just as a method of selling ads seems to have been a complete bust so far. If that is wrong you know better, please link us up.

I'm suspicious that their predictive power to sell ads actually works for the people who buy those ads but I guess we aren't likely to know for sure. I do wonder "who dominated their industry segment in sales by being an early adopter of google ads" I don't know anyone. It's not a great metric but what else do we have?

They train winning code from scratch, I think, so you wouldn't have been able to just train the model with the "unseen" data.

They scraped data from the website of the org that wanted the results and funded the competition. That data was supposed to be unseen for the competitors and used to grade the models. This cheat was to use that scraped data in its training set and, beyond that, hard code some predictions.

It's looking up the answers in the grading sheet while taking an exam.

Your parent is saying kaggle takes the training code and runs it themselves. This makss secretly training on the known right answers impossible.

This competition allowed submissions to include extra data files that can be used by the model. The cheaters added a file with data from another website that seemed innocent, but secretly encoded extra information (perfect answers) in IDs. For 10% of predictions, the code via a set of obfuscated operation retrieved this information and presented it as the answer.

There is no such thing as "training code".

You have a model that you have trained on some provided data - the training set. You give kaggle this model. Kaggle grades your model on some different data your model has never seen. The better your model classifies this data it hasn't seen the higher it scores and the more money you win.

So again if you trained your model (code) on a training set that you have illegally obtained. That is broken the rules of the competition to get the additional data. Data that is also going to be used for competition verification and grading, then you have cheated. Doubly so if you just hardcode outputs to boost your score, which they did here.

They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught.

It really is equivalent to looking up the answer sheet while taking an exam.

People are saying that you do not just submit your trained model to Kaggle. You also submit the code that was used to train the model from the training set, which is used in the winning models to train them from scratch on the training set. Of course, that wouldn't have prevented this type of cheating of course, but it does mean that you can't submit a model that was trained on your own private data set.

You can overfit a model to your hold out set quite easily with repeated trials. It's a trap you have to avoid in normal circumstances! (Feynman: "You are the easiest person to fool"). Even if you have to submit code to generate your model parameters from "the training set" (which hasn't been explained at all well by "People" if that is indeed the case) you could do that overfitting deliberately here with the illegal unseen data as your hold out set. Aside from the advantage of a bigger training set. Aside from the advantage in model selection, which is not done with code from a training set. Aside from the advantage to your feature engineering also not done in code. Aside from the advantage to your regularization choices, bias parameters etc etc.

So yes you absolutely /can/ submit a model trained on your own private data set even if what you submit is a model code that will be re-trained. Even if "the training set" is different to the provided - you still have that scraped data so you can slice it up with the provided training set so that any selected training set does well against the rest. Now the overfit you've just carefully engineered should win against the honest models unless you suck, right? It's kind of risible that they had to go further and hard code certain results, don't you think? Perhaps if they still couldn't win with scraped, illegal additional data then everyone else had illegal data too? Perhaps Kaggle is not a good indicator of how good ML techniques are in practise? Perhaps Kaggle systematically overstates ML effectiveness due to this kind of uncaught cheating in many of their competitions? I bet kaggle won't look too hard at that.

> There is no such thing as "training code".

I'm by no means an expert in ML, but my understanding is there's some code that is run to train the modal. I meant that by "training code". My regrets if my terminology was unclear.

> They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught.

No, this is wrong

I suggest you look at one of the other comment where people have explained why this is wrong. They did a better job than me.

https://news.ycombinator.com/item?id=22124760

https://news.ycombinator.com/item?id=22126489

https://news.ycombinator.com/item?id=22124193

No it isn't. See above. Overfittiing to a holdout set is so easy and common you usually need to take steps to avoid it. See my comment directly above. Best.

Thank you. I found your comment extremely helpful.

Reply on news.ycombinator.com