Pragmatic Theory, Commendo, Yahoo and AT&T hit the Netflix grand prize
netflixprize.com
netflixprize.com
In any case, I'd love to see Netflix start another challenge round. A million dollars isn't a whole lot of money for them, it brings in a lot of publicity, and provides a 'fun' way to get some research done in the data-mining field.
http://www.netflixprize.com//rules
To qualify for the Grand Prize the RMSE of a Participant’s submitted predictions on the test subset must be less than or equal to 90% of 0.9525, or 0.8572 (the "qualifying RMSE"). After three (3) months have elapsed from the start of the Contest, when the RMSE of a submitted prediction set on the quiz subset improves beyond the qualifying RMSE an electronic announcement will inform all registered Participants that they have thirty (30) days to submit additional candidate prediction sets to be considered for judging.
This starts a 30 day period during which all the teams can submit their final versions. After that, I believe Netflix will run the algorithms against the unreleased set of movie rating data.
The final call announcement would not go out unless a team beat 10% on the test set. That announcement went out at 4:30pm PST today.
It is still quite possible that the entries can be rejected if the reviewers find the system description lacking or if we are unable to reproduce their result. That clock starts after the 30 days runs out.
Is it possible that there is over-fitting going on?
it's rather important to us that the solutions aren't over fit so we've worked very hard to make it hard if not impossible to do.
I think it's safe to assume that we wouldn't have used new algorithm(s) if they didn't work well on our ~3B ratings.
Probably no one would have won this contest if it wasn't for the fact that many of the top teams published their results, and so there was cross-pollination of ideas. Pragmatic Theory didn't contribute in this manner. Kudos to them for making I think a significant improvement on the best of the published models however.
An open source version would be a significant act of charity. It would be nice to see older versions or versions that were crippled enough to be education without watering down the reward that the winnners receive.
Open sources projects have already been created in Python, Java, C, and C++. None of these projects include the full algorithms from the 2007 or 2008 papers.
The projects can be found from links on the Netflixprize forum:
Actually receiving the prize will stimulate more people to do ML research. I would be happy to see some other AI competitions similar to IOI, IMO or TopCoder with big publicity.
Netflix are pioneers in such challenges and have the opportunity to be some of the founders of such future competitions.
Has Netflix announced any follow-up challenges?
In all seriousness, can anyone enlighten me as to who owns the submissions now?
Technically they are obliged to license them only if they are about to receive the prize.