Netflix Prize So Close, They Can Taste It | Hacker News Reader

Netflix Prize So Close, They Can Taste It

wired.com

63 pointsnreece16 comments

Surprising that more companies haven't tried to replicate this - there's no way you could get so many qualified people working on something for three years for a million all up as employees. Not to mention the publicity.

I can't think of a lot of problems that would work with this kind of contest. It has to be hard enough and interesting enough to be like a game for the world's smartest people, you have to have a prize big enough to make it not an insult, and you have to have an existing solution that is compelling and interesting to talk about. And, of course, you need a dataset so people can know when they've reached the goal.

I can think of only a few companies that could do something like this. Google, Microsoft, Yahoo, are obvious choices, since they all tinker with recommendations and other such things for search, videos, and shopping. Apple iTunes, eMusic, Amazon for their stores. What else would work here? A lot of startups are playing in this space, often founded by PhD's, but they don't have the $1mil or the big and interesting dataset and pre-existing market-tested solution.

I dunno, I think there are plenty of other areas for some competitive CompSci.

As a random idea: Oracle puts up $1MM to increase their query optimiser by 10%. Plenty of test data there, easy to measure performance gain through a decrease in CPU / disk IO. And lots of gnarly edge cases. The upshot to the economy in saved cycles could be huge.

Or a transcontinental shipping company could run a competition to improve their prop design, or engine design, or their hull design. Fluid dynamics is a challenging, computationally intensive field. And any percent point improvement on these makes real savings.

Or a city might put some real smarts behind their traffic light system, and intelligently improve traffic flow. Are they much more than day-parted timers and queue counters today?

I think there's plenty of room for other competitions / grants like this :)

Testing an improved Oracle query optimizer would require terabytes of different databases in order to have a representative sample of real customer databases. That's not really practical today.

If a contest participant comes up with a proposed design improvement for a cargo ship prop, how do you actually test that? Testing in a CFD modeling tool is one thing, but to really be sure you have to build it and put it on a ship.

For traffic engineering, testing is the real problem again. Who is going to write an accurate simulation of an entire city's traffic flow so that contestants can use it to test their light timing algorithms?

By contrast the Netflix prize involves a small data set and doesn't require building anything in the real world.

First, I don't think that most companies have even harvested the low-hanging fruit in their data. They need to do it before they worry about optimizing it.

Second, there really are a lot of barriers to this. Your idea about improving prop design is cool, but it requires more resources than a few buddies can assemble: huge computational horsepower, as you note, as well as the ability to actually fabricate and test the props.

Liability concerns will squash much of the innovation. Your traffic flow question is really interesting. But I don't think I'll be entering if there's any chance that some lawyer is going to add me to the list of defendants when someone gets killed by a driver running a red light.

In some areas the government outright forbids private development. (I don't mean to turn this into a gun control debate; for good or bad, this is just an example) The USA used to be the world's innovator in firearms design. People like John Moses Browning are part of the reason that our military became the class of the world. But starting with the 1938 NFR, taxes and regulations made it impractical for private individuals. The result is that now improvements are really only the result of direct government funding (e.g., Armalite and the AR-15), or occur overseas, like in Israel. Similarly, I imagine that any biological or pharma work would wind up with the FBI at your door (assuming that you could get by without million-dollar hardware anway).

So, you definitely have a better imagination than I for possible places to use this development model. That said, the areas you've proposed, as others have mentioned, are bigger problems to solve, sometimes much bigger. They require a lot more capital-intensive pieces to come together, or they require accurate models that don't necessarily exist.

Of course, that didn't stop companies from competing for the X-Prize. So, perhaps in the areas you suggest the prize just needs to be bigger (maybe a lot bigger). So, while an improved query optimiser for Oracle might require hundreds of thousands of dollars worth of hardware to develop, a $10 million prize would still be sufficient to interest some teams. At some point, the line is crossed between what is a good development+marketing investment, and what the company could do with its own in-house developers. I don't know if $10 million is past that line, or if it's enough incentive to bring out the geniuses in the field (the ones that don't already work for Oracle).

Oracle would also have the problem that they'd have to create some sort of stripped version of Oracle that only works on this one data set, or something, because otherwise they'd be giving away the biggest license of their software to any developer that signed up (because to test major improvements to Oracle, in the places that matter to Oracle, you need to be running it on big MP machinery with terabytes of data).

So, maybe that's why more companies haven't done it, despite the apparent cost-effectiveness and marketing value (NetFlix has probably gotten a million bucks worth of tech press out of this contest). The cost of getting people on board might be much higher, and the cost of providing the tools to contestants may also be dramatically higher. There may also be negative media consequences. NetFlix spun it beautifully...such that we think of them as very smart scrappy guys and hard to beat, but also eager for a challenge from upstarts. Maybe a big behemoth like Oracle would come across as, "Oh, now they can't even be bothered to build their own software, and expect a bunch of Open Source experts (since Open Source is where a lot of the expertise in databases currently exists, outside of companies that wouldn't want to help Oracle) to do it for a pittance." So, this kind of thing might turn out to be negative publicity for a big company, where it is great publicity for a small company like NetFlix.

Just some devil's advocate stuff. I think you're probably right that there are other areas where this would work. I don't know that you or I have thought of the best examples of them, though. Maybe someone reading this thread might have a company for which this model fits perfectly, though.

Good to see 2 engineers from Montreal taking the lead! It's great that Netflix amended its rules to allow Quebec residents to participate.

I find it interesting that there seems to be an asymptotic increase in difficulty right around the 10 percent mark. Did Netflix estimate this before they created the Prize, or did they think it would be easier (or harder) to get there--perhaps that they would have gotten there already?

There has to a be a limit to the amount of data that can be taken from what is provided to improve the effectiveness of the suggestions, I'd say there close to the point in which techniques to make sense of the data have reached there limit.

The system can only be effective to a point because of the unpredictability of a user.

I was psyched about this marketing technique, and social design project from the get go. Even though it's not quote massive collaboration, there is a powerful search implementation being leveraged in having many problem solvers hitting it from different directions simultaneously. Fascinating if this technique can be applied to other problems (akronim totally agree). I wrote up something back in April (but the Harvard group did a better job). The Big Shift (Harvard group): http://blogs.harvardbusiness.org/bigshift/ just check out their many posts on collaborative design

My link:http://www.victusspiritus.com/2009/04/25/the-search-power-of...

It's surprising that 10% turned out to be such a good target. If it was 5%, it would have been done almost right away, and it looks like 15% might be impossible.

Clearly the Prize has been a great thing for Netflix, in several ways. What will they do as a follow-up, once the Prize is won, and the press dies down?

I'd up the number and start it all over if I were them.

I'd love to see them add another data point to the next challenge. We have user ratings & date; when teams actually started utilizing date instead of ignoring it like Cinematch the scores shot up significantly. Imagine if they added something else to the dataset; perhaps movies that have been "viewed" and/or "queued" by users but have not been rated or something more off-the-wall like some basic demographic data or what level of Netflix plan they subscribe to.

You have a great point there... personally, when I used to subscribe to Nexflix, I wish there was a button that said, "I've seen this movie". That way, they could know which movies I've seen (and like/dislike) outside their system.

5 million for an additional 5%? Something like that, you mean?

Reply on news.ycombinator.com