I'm curious how you quantify that?
Not picking on you at all, just going on a tangent because I've been thinking recently about the good ideas that end up on the cutting floor because their impact, while positive, is intractable or completely impractical to quantify.
Some ideas (like this Netflix one!) are "obvious" winners, because they have a diffused, positive impact in many important dimensions – each of which is almost impossible to measure, but the integral of which is almost certainly greater than the idea's cost, probably by one or two OOM. If there's high conviction in an idea being a 10x idea, and attempting it costs quite little, it's better to do it now and maybe consider measuring it later. Who cares if it's 9x or 12x ROI, the point is it has a big margin of safety and large expected returns on capital.
But in a "we can't greenlight this project unless we can directly attribute it to positive motion in our KPI" org, these flavors of idea are dead on arrival. The double-kicker is that the cost of trying to measure these ideas is often greater than the cost of the idea.
For some ideas, especially when it's a rounding error on the company's annual budget, it should be OK if we don't invest much effort into quantifying the results. It's one of those rare set of ideas where it's anathema to the professional/investor culture of SV of the last decade, while simultaneously being how plucky Seed/Series A companies can punch above their weight.
Now, I think it is fully fair to say that this was very cheap for them to do. In which case, why not do it? But I think you would be hard pressed to give me a counter factual world that is believable where not doing this had a meaningful impact on Netflix's future.
I can certainly mostly agree with this. I just also think them producing their own shows was likely more impactful, and had basically nothing to do with this.
Don't get me wrong, plenty of moves in life are "win more." Such that I could see this one fitting that description. But a lot of "win more" choices are surprisingly low on the ROI side, when you control for everything else in that situation.
Of course making their own shows was more impactful in a bunch of ways. But, the investment is also so much more. A season of a show is in the tens to hundreds of millions.
But, the real difference is that Coke is, in fact, selling their own brand and products. For Netflix, they are largely selling access to other people's products. In a very real way, Netflix found an arbitrage opportunity for viewing movies. With how poorly the content owners calculated things, they got good deals at the beginning so that they could do well on their investment in the CDN infrastructure so that they could deliver a good experience to users with minimal unit costs.
This is, in a large sense, why Netflix is having to raise prices now. The content owners are starting to demand more money for that access, driving up their unit costs. Combined with a draught of original content, expect that they will lose access to their own productions in time. You can already see this with the likes of Daredevil.
My comment about Coca-Cola was to draw the analogy that just because they don't have trouble hiring (which you mentioned) doesn't mean it can't be helped.
If you saw a measured rise in hiring, I will trust you on that. My assertion here is largely on the trajectory that Netflix seemed to already have and how there is no clear "inflection point" in any of their numbers post this contest that I have been presented with. And probably more cynicism than makes sense for how little impact ML seems to have on any choices that Netflix seems to show.
More directly, though, if you were to plot subscriptions to Netflix over time across an unlabeled chart, would you be able to see this contest?
Take my challenge. Plot their subscribers and revenue over time, hiding the year/month. See if you can spot any sort of inflection where this happened.
It /could/ be that they would have seen declining numbers absent this. Such that this prevented a bad inflection on the chart. I don't know that I find that compelling.
Like, I get that you can't measure everything. There are plenty of reasons to believe that.
I just can't bring myself to accept the odd logic that "popular tech company got press coverage which was instrumental in their growth." Seems there is a circular logic at play there that makes this such that, even if it is true, it is not actionable for anyone else.
Put another way, popular things get press coverage by virtue of them being popular things. At large, as long as they don't implode in very terrible coverage, they don't have to do much to get more coverage. Just doing anything will do that for them. To that end, odds are high we would still have discussed this even if they only offered something silly like a lifetime subscription for the winning team.
Some scale for perspective: if this had a 10x ROI, they'd get $10mm for $1mm. At EOY 2009 (same year this competition was paid out), Netflix had a twelve-month-trailing Revenue of $1.671B. In 2006 they took in ~$1Bn. So it's 0.06% of annual revenue at payout time, 0.1% of annual revenue at announcement/green-light time.
The idea had a large margin of safety: in the worst case, nobody pays attention and you've generated some IP which would have cost ~$1mm to generate anyways, and which would theoretically increase retention (a top 3 KPI) by X%. And it had big upside potential, in the form of additional marginal hires, press/exposure of the core product in the news cycle leading to marginal subscribers.
Re hiring: I think it's totally plausible that it contributed to ~5% of 500 peoples' employment decisions since 2009.
Re subscribers: it's plausible to think it contributed ~1% to 10k peoples' decisions to subscribe, most of whom have retained to-date. It's classic "Brand Advertising" – and although it's difficult to quantify, it works.
Those numbers could be way higher, but it doesn't really matter if they were or weren't, the margin of safety, uncapped upside of the brand advertising angle, and low relative cost made it a bona fide good idea.
Edit: FWIW, that we're still discussing this idea in 2024 is more evidence that it was a good idea. This whole thread is full of Netflix's target demographic. We're all talking about how well-compensated and smart Netflix engineers are... You would never be able to purchase that kind of long-burn, compounding, high-expected-value attention for $1mm from any advertising network, in 2006-2009 or now.
I think I would push on the ~5% of 500 people's employment decisions. With the sizes of the teams that competed, I'd not be too surprised that the teams involved were largely the extent of who was directly impacted by this contest. Everyone else was as happy as those of us that read any of the blogs/articles that they would publish about their infrastructure. Which is not nothing, but probably hard to spot the difference. (Ironically, I would argue that the raised morale of the people that publish the articles are probably more measurable? Which, I confess I don't think is a consistent position from me.)
And I stress that I agree it was a safe thing to do. I am even glad that they did it. I just don't know that you could really convince me they would not have had the revenue that they had without doing this contest.
$1m to generate some publicity and attract talent? Sure. They're probably paying headhunters similar amounts in comissions each year.
More to the point, though, this is a bit of a tangent/deflection from my question. How do you assert the return on this $1m? This line simply says that no return was needed for it to be spent. It shows nothing to say how you calculate what is made from it.
It was in the news a lot, and was discussed on a lot of tech sites. Plus it gets people talking about their recomendation algorithm, and makes people thing Netflix subscription is more valuable becasue it recommends good shows. It wouldn't be cheap to get the amount of media that they got through more traditional marketing.
The main one I don't think I would have doubts about is the recruitment. But, I don't recall them being a place that needed recruitment help, even at that time.
To be clear, I found the thing fun to consider. I certainly am not upset that they did it. I do harbor a gut feeling that its ROI is greatly overstated.