Comment by rm999 | Hacker News Reader

Comment by rm999 | Hacker News Reader

Kaggle is a great idea; I actually entered the industry partly through a data mining contest (pre-kaggle). But I think the article overstates Kaggle's influence on industry, at least at this point in time.

>the Kaggle ranking has become an essential metric in the world of data science. Employers like American Express and the New York Times have begun listing a Kaggle rank as an essential qualification in their help wanted ads for data scientists

No, it hasn't, and the nytimes job posting they link to doesn't list it as an "essential" qualification (they didn't link to the amex posting). I know many people with experience in the data science space and very few of us have taken part in a Kaggle competition. It's not that there is anything wrong with Kaggle, but the pay is low for the required effort to differentiate yourself. Many modeling competitions I've seen require an inefficient use of time in the "diminishing returns" part of the process, which means winning requires a lot of free time. I worked with a guy who won a couple prominent data modeling competitions, and frankly I thought he was a mediocre data scientist (but a very hard worker).

I sometimes wonder who takes part in the competitions; and then I remember myself from six years ago, applying for jobs and looking for a way to make my resume stand out.

Agreed. In conversations with data scientists looking to hire, we've never discussed Kaggle aside from "wow they reeeeeally focus on that evaluation metric."

In fact, our conversations quickly turn to the Netflix Prize, where first place won the competition with an algorithm that could not be ported to production, and we discuss how poorly these competitions map to reality.

None of the data scientists I know hire based on Kaggle score. Several don't even think positively of Kaggle.

Why couldn't it be ported to production?

If I recall, BellKor made many, many models based on Gradient Boosted Decision Trees, Restricted Boltzmann Machines, and kNN. They tried many different feature subsets, added temporal weighting, and tried many reduced-dimensionality representations (SVD, NMF). They then stacked them all together into one final ensemble whose RSME beat everyone else's on a hidden validation set.

In a production environment, this is probably an insane amount of transformation, feature extraction, and classification for marginally little gains in precision (as defined here). But I'm only a year or two in to building production-environment classifiers, and nothing at Netflix's scale (though not tiny either-- it is a problem if I can't do feature extraction and high-precision/-recall* classification within a few milliseconds).

* - mid 90s, for a hard NLP/social graph problem.

It's funny to me that people are so quick to poo-poo the complicated modeling done for the Netflix prize. When did production-worthiness become the only important thing? It's like saying Watson was useless because it can't play a concurrent game of Jeopardy with thousands of people on the web.

Just like in research, it turns out that relaxing real-world constraints on a problem is often a great way to make progress. I would not have to search long or far to provide much worse uses of million-dollar grants/projects/big-data-software.

Don't mean to poo-poo. I love scrambling for models and methods. In the latest problem I worked on, I did the same thing. Implementing papers, trying to avail myself of semi-supervised or reduced-dimension representations, tweaking models and features every which way... it's illuminating work, and good things come out of it.

But, in the end, companies like Netflix or where I work are immediately looking for cheap ways to make X happen easier, better, and cheaper. But then hopefully the smart papers go on a shelf or are easily Googleable, and the rest of us get to learn from their efforts.

I can fit long-term and short-term goals in my brain, too, mister.

Largely what textminer said. In fact they didn't use anything from the grand-prize winner because it would require too much engineering effort and because their concentration on the problem shifted. They did however use the two best methods from the "progress prize", so I wouldn't call the contest a failure.

>If you followed the Prize competition, you might be wondering what happened with the final Grand Prize ensemble that won the $1M two years later. This is a truly impressive compilation and culmination of years of work, blending hundreds of predictive models to finally cross the finish line. We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment. Also, our focus on improving Netflix personalization had shifted to the next level by then.

From http://techblog.netflix.com/2012/04/netflix-recommendations-...

I am a 'Kaggler', I have done well (prize winner) in some past competitions and I could not agree more.

In the last week of a competition, you go into kitchen sink mode trying anything and everything to squeeze the last few points out of your models. The objective of the competition from the competitors standpoint is to win, not create a solution that could be deployed into the sponsor's production environment. On going maintenance of the model is not a consideration.

As far as the netflix competition goes though, the final solutions did help publicize the potential of ensembling and RBMs - I am grateful for that. My personal approach to the competitions is to use them as an opportunity to try out new modeling techniques (i.e. RMBs, deep beliefs etc...) on real world data. It has the added bonus of potentially paying off with prize money (as long as you don't get screwed[1]).

I would like to hear from past Kaggle sponsors and see what they have done with the winning models.

[1] http://www.gequest.com/c/flight/forums/t/4284/acknowledging-...

Reply on news.ycombinator.com