Comment by mwexler | Hacker News Reader

Comment by mwexler | Hacker News Reader

This stuff is great. The rewards/prizes for the best model are so minimal compared to what it usually costs to build a great model via a consulting contract or hiring a high quality miner.

Similar to Mechanical Turk, we've managed to create a completely different value structure for some amazing work by smart folks... mostly by making it a competition. Great exposure for winners, sure, but these prizes are pretty minimal.

http://www.kaggle.com/c/GiveMeSomeCredit, for example, has a total basket of US$5K (only US$3K for first place) for a model predicting credit scores (in this case, likelihood to default or have financial distress). Folks I talk to who do this type of work professionally tend to charge far more than that to create these models.

For the company sharing this data, of course, big win: They get a cheap, potentially fantastic new model, and the creator gets some good exposure and some cash. But if these take off, they can really change the economics of how this work is created.

You are right -- there are some neat aspects to this model.

However, it also tends to devalue the work invested by the analysts. They are doing the work for essentially a lottery ticket -- winner take all. That's the reason that it can be so cost effective to the company running the competition -- they don't absorb the costs of the failures (or less optimal approaches).

Those costs have to be absorbed by someone. In this model, they are eaten by the analysts, who (generally) don't have enough resources to cover those costs. Due to that, I don't think this model is sustainable or will really catch on.

OTOH, in some (ideal) academic endeavors, having multiple groups compete for more funding or for a prize has certainly benefitted the sciences. In that case, however, the competition was more friendly than zero sum. Also, I believe the different sides tend to share information more than hoard it, yielding lessons learned to the entire group from one team's failures.

That's fair. I originally had the word "devalue" in the comment, but I pulled it out. Why? No one is forcing the analyst to participate, but instead, they absorb the costs to get exposure or growth (or fun). Whether that's a fair tradeoff is up to each person, but given the number of folks participating in Netflix competitions, these, and others out there, a group of smart, analytic types feel the tradeoff is worth it.

I'm not (yet) a highly paid consultant in this area, but I still don't think it's worth my time to participate in most of these Kaggle contests. Even the $3 million prize for the healthcare prediction isn't worth it if you think you have a chance of beating current best-practice (and think you can sell your time or have a friend who can sell your time to corporations).

For the Netflix competition in particular, a lot of the entrants were being "paid" in research papers, grants, and academic salary, because it was a high-profile competition in a theoretically not-entirely-settled area (collaborative filtering), so even many non-winning entries could get published papers out of it. I don't think that can be infinitely replicated, in part because once there are dozens of contests, it's less likely any one will be as high profile, and in part because not every ML contest is as academically interesting (the Netflix setting was sort of "weird" from the perspective of traditional statistical models, not being a straightforward regression problem, whereas some of these are pretty straightforward).

My experience is the opposite. I entered a number of competitions last year. For a data scientist, the opportunity to access clean data sets and interesting real-world problems, as well as to see how your techniques compare to others', is really compelling. (At least it was for me!)

I won a couple of competitions, and discovered that the competitive environment pushed me to create new algorithms and develop new ideas that I otherwise would not have. (Normally I would have thought my initial answers were "good enough" - but in the extremely competitive environment of Kaggle that is never true!)

And of course there were many more competitions I did not win. Honestly, I learnt much more from these - because by the end of the comp I knew quite a bit about the problem domain, and had tried a few ideas out, so when I then read the winning papers it gave me heaps of new ideas and insights that I could use in future projects and competitions.

I became so interested in the company that I invested in it, and then started helping out here and there, and finally joined full-time and this week have moved to SF (from Melbourne, Australia) and am now Kaggle's President and Chief Scientist.

It depends on what you mean by "sustainable." Is it going to create a sustainable business model that will displace the whole analyst labor/consulting market? No. A significant portion? Probably not. Will it create opportunities for firms needing analysis to get it done cheaply and for non-traditional or inexperienced analyst to show their stuff? For sure.

Might it lead to totally new ways of approaching problems that would never have been discovered otherwise? Most likely.

So it's not going to replace analyst in a niche that is already hot, but it will be a valuable component of that ecosystem.

I wish them the best of luck and I'm already thinking of ways my company could use this.

Reply on news.ycombinator.com