Netflix’s Secret Special Algorithm Is a Human | Hacker News Reader

Netflix’s Secret Special Algorithm Is a Human

newyorker.com

214 pointsdonohoe59 comments

Hollywood Video's corporate office had a guy, one guy whose job it was to just watch movies. He'd watch them, make sure it was categorized properly, then create associations with other movies that "you might like" if you like this movie, and vice versa. All kept in a spreadsheet that we later plugged in.

We also thought there should be an algorithm, but he was pretty dang good at what he did.

It's simple ROI. There is not that many movies this is a task that one doesn't need to scale with computers. Hiring a guy is likely much cheaper than investing on the software. Plus I am sure there are plenty of people who would like to watch movies for their day jobs.

On the other hand, I am pretty sure there is at least some degree of automated video classification at Youtube.

I think it does need to scale.

What happens when after a bus event? Someone else has to watch every film in existence? It is not enough to just watch the new films - you have to have a memory of all other films in order to make that association.

Seems reasonable to me that you could have a team of categorizers/critics.

That'd be several times more expensive of course, depending on how many people you added, but I think it also might improve the results if the team was picked well. Finding associations between movies is probably something a group of people can do more effectively than one person, since recall will be better.

There might be an issue of disagreements within the team, but I think at least finding associations between movies would tend to be fairly non-controversial. We might disagree over whether or not The Italian Job is a good movie, but we probably both agree that it is a heist movie.

What about aggregating movie reviews and classifying movies based on their reviews. Would be an interesting problem regardless of accuracy.

> Someone else has to watch every film in existence?

To some extent, all the film program students and film critics provide your backup reserves: they're watching tons of films on their own, and you don't even have to pay them until you hire them.

> VidArc was a tiny store in a mini-mall, with barely room to squeeze past other customers, but it made up for its size with the percentage of rare and obscure titles that were available, and with the knowledge of the film-nut staff, notably the cinema-obsessed and mile-a-minute talker Quentin, whose low-budget life at the time has been explored in several books. Denise's card was number 1410, and when I made my trek from Oregon in the Orange Monster (my '72 Olds Cutlass), I became a customer as well, discovering the world of strange and disturbing cinema under the tutelage of Quentin, Rowland Wafford, Gerald "Big Jerry" Martinez, Stevo Polyi, Roger Avary, and the owner of VidArc, Lance Lawson.

http://toddmecklem.com/quentin.html

    I am pretty sure there is at least some
    degree of automated video classification
    at Youtube.

there's definitely collaborative filtering.

This reminds me of Derek Sivers' album recommendations for CD Baby:

http://sivers.org/hi

Basically Sivers just listened to every album and did the same thing, and when he couldn't get to all of them he hired someone to do it full time.

How many hours of new programming do they add a day? How many people would it take?

Very interesting. Did he enjoy it or did he get bored? Did he have to watch some horrific films? Did he feel like his brain was being polluted as more degenerate films come out? Did it affect his perception of reality?

Just interested.

Only a few times have I watched a full three movies in a day, and I found it exhausting. I think having to watch movies all day every day would be much harder than it first sounds.

Same here. Watching the Star Wars trilogy in one day was exhausting, and also perhaps a bit sad. Watching the entire 6 films would be killer.

Spare a thought for the people at the BBFC in the UK who in addition to having to watch every feature film to rate it also have to watch all the pornography videos to rate and check for legality.

I attended a university debate once between a BBFC censor and a female porn director. Watching the two of them talking about "counting knuckles during a fisting scene" to check that the UK requirement that at least one knuckle is always visible was a truly bizarre experience. The BBFC man had the look of an individual who has stared into the heart of darkness for a little too long. He had some excellent stories though!

If anyone does organise debates, talks, conferences etc in the UK on associated topics I can highly recommend getting in touch with the BBFC. They're really keen to engage with the wider public rather than just be two signatures on the screen just before a film comes on.

Have the BBFC ratings dropped though? I mean, stuff that was horrific and rated as 18 in the 1970s is now routinely part of films rated 12 or 12A. The sheer amount of violence that is part of films has significantly risen.

That is basically what Netflix does. They have teams of people trained to watch movies and tag them with lots of metadata -- really precise metadata. Then the algorithms take over and do the actual recommendation.

http://www.theatlantic.com/technology/archive/2014/01/how-ne...

In other realms, we don't call this data but intelligence. Humans will still make the decisions, but in more and more places, the decisions are more likely to be informed by data (intelligence).

Consider that Netflix's data is tiny compared to the amount of data that any government must sift through. Algorithms don't make government decisions, people do. But they (hopefully) base those decisions based on intelligence (some of) which was gathered and filtered by algorithms, then synthesized by humans.

I am not entirely sure what you are saying here, but the way I have always understood it is: data is knowledge; the ability to apply knowledge is intelligence.

Netflix has a considerable amount of data (knowledge) and its algorithms exemplify some efforts to apply that knowledge (intelligence). As it stands presently, though, humans still tend to be more intelligent than any algorithms we have created. (Generally speaking, of course.)

I think we are trying to say the same things here, right?

I assumed they were using the other meaning of intelligence, as in "signal intelligence"

I'm trying to reframe what Netflix does as something that is already done - we just tend to use different names for it in those other domains. What's new is that we're doing this old thing - processing massive amounts of data, extracting the relevant facts, synthesizing those facts into a coherent story, and presenting that story to decision makers - in new contexts.

In the Netflix context, what is mostly called "data" is called "intelligence" in, say, government decision making.

This is another facet of the "amplified-teams" trend that's been happening over the past few years. This book review of 'Average is over' has some good information about it:

http://ieet.org/index.php/IEET/more/searle20150109

"In his vision intelligent machines will revolutionize everything from medicine to education to business management and negotiation to love. The human beings who will best thrive in this new environment will be those whose work best complements that of intelligent machines, and this will be the case all the way from the factory floor to the classroom."

Very interesting times ahead.

Perhaps another way of stating that is: The people who will best thrive are those whose work cannot be automated. Which, in some ways, has always been true.

In Health Informatics at least this is taken even further

    Data
      X of Y people have some disease
    Information
      Based on data, I can predict disease likelihood
      given some environmental and personal factors of
      a patient
    Knowledge
      Using informed predictions, I make good inferences
      about how to proceed with diagnosis
    Wisdom
      Using knowledge and experience I choose the right
      approach for treating and diagnosing a patient
      which is efficacious, healthy, and works with the
      patient's actual needs

It's easy to draw these lines in other places or to call the tower a lot of woo woo able to be reduced into inferences atop raw data all combined correctly... but it serves to remind just how difficult it is to combine the right data in the right way to make the right decisions at the right times.

It also serves as a sharp counterpoint to the idea of, say, machine learning patient diagnoses. It turns out that diagnostic accuracy is terrible, but not because people are directly bad at it (even if they are) but instead because knowledge/wisdom dictates that perfect accuracy isn't that valuable---perfect care is and that can involve chasing down treatment and care avenues that would never be predicted or acting on information that is not currently in your model.

As much as people (especially here) think that everything can be solved by Big Data/Machine Learning, human experience and intuition are still very important.

No, you can't A/B test your new logo, or the design of your page from scratch.

Big Data can't tell you your product sucks. Option A may be better than Option B but this is in the context of both options (and not considering all other possibilities)

I see companies firehosing every tiny bit of consumer data hoping to be able to make sense of it all and find something there. Meanwhile they're missing whatever their competitor is doing and what their consumers are liking about it.

Those companies are just cargo-culting into the Big Data cult, like everyone else. It doesn't mean they aren't getting useful answers from their black box.

Heck, sometimes it's even useful to have a Magic 8 ball make a decision for me.

See also Shazam's Secret

> The hunt keeps Mr. Slomovitz on his toes. Every morning, he skims dozens of music blogs, checking for new releases he might have missed, as well as the iTunes, Amazon.com and Billboard charts, and blog aggregators like the Hype Machine.

Most weeks he also goes to local record stores to see if there is something in stock he has not heard of, or if older albums are being remastered or reissued. And he listens to local radio stations, especially near universities.

"Shazam’s Search for Songs Creates New Music Jobs" http://www.nytimes.com/2011/02/14/technology/14shazam.html

Shazam actually has 2000 music geeks in a call center who listen to tiny audio fragments and guess which song it is. The algorithm just makes sure the rock geek gets the rock songs, and so on.

Really? I'm genuinely interested in this. Do you have an article or interview or something with more info?

I see this argument akin to something along the lines of, "A website is really a human because someone needs to maintain it and make decisions on it's appearance."

You are describing a tool. People don't say hammers are humans, either.

Netflix's special algorithm is indeed an algorithm.

I was really hoping Netflix had a secret warehouse filled with hundreds of workers whose job was to provide recommendations based on what you like, a la 'invisible boyfriend' [1]

Sadly, they just have people on the quest for the perfect hammer.

[1]http://www.washingtonpost.com/news/the-intersect/wp/2015/01/...

It always amazes me how the people in the content creation industries keep their jobs. Even though comprehensive studies have shown that the public responds randomly to entertainment products, they still skate by claiming to 'know what the public wants.' Analysis of the decisions of past executives at movie companies showed that executives who were said to be "on a hot streak" were mostly benefitting from the results of projects started by their predecessors, and then when their "streak" ended and they left, their replacement got the same benefit from the projects they left in the pipeline.

The book 'A Drunkard's Walk' analyses various different industries and situations and shows where randomness shows up. The success of media is one of the strongest ones. No factor correllates with success. Not budget, star-power, genre, directors, nothing. For every runaway hit there are exactly as many abyssmal failures. Before Titanic hit theaters, film critics and industry insiders were dead certain that it would prove to be the most monumental theatrical failure in history, making Kevin Costner's Waterworld look like a walk in the park. Of course, they were wrong. They will always be wrong as often as they are correct. Their entire careers are built on, quite literally, absolutely nothing. These executives choose what gets produced, and they are incompetent at it. Yet they are paid millions of dollars. It's astonishing that they get so far with unmitigated bullshit.

This is why large movie production houses will die. Their performance has always been random in terms of producing successful content. But they always made up for it by having total control of the distribution. Now, distribution is worthless. Any 12 year old with an Internet connection can distribute media better than large corporations can. Take away that control and profit from distribution, and those companies will end up simply fading into bankruptcy after enough failed projects pile up.

You might also be interested in Duncan Watts' book "Everything is Obvious", which explores this tendency we have to assume that observed success equates to some skill. Its a nice read.

Large movie production houses won't die in the near future, because predicting a financial winner isn't as hard as you make it sound. We currently have a movie economy that is driving the big producers to make big budget comic book sequels that do well internationally (generally based on the universal appeal of liking suspense and action). They've boxed themselves into a very profitable corner of funding high budget movies that independent producers can't fund and then profiting internationally. They make a movie for $200M then make $300M+ in sales.

I do agree that more independent (but still highly funded) producers are rolling dice, then manufacturing winners with their marketing power.

I see how netflix's dataset helps them produce an addictive serialized drama like House of Cards. There are many examples to learn from, and the shows are basically built out of a formula.

What I don't see is how this would help you make a critically well received documentary about Nina Simone. Sure, you might be able to predict a lot of people will stream it, but how does that make it well received at Cannes? Nothing in their dataset is telling them about the art of film-making.

No, but their data allows filmmakers to gain a better idea of what the consequences of certain decisions are likely to be. It's not so much as allowing algorithms to dictate creative decisions as it is enabling those decisions to be better informed. As for targeting film festivals in particular, I've no doubt that Netflix has sufficient data to look at sort of festival-goers (and those like them) in particular. While it can't provide a guaranteed formula for festival success, it should still give a filmmaker another tool that helps them do their work.

The argument of the article is that Netflix's dataset doesn't really help them make decisions like what the content of their next documentary should be. It argues that what happens instead is that Netflix's programming honcho, Ted Sarandos, picks projects based on the person who's putting them together, looking for creators with a solid creative track record and an existing cult following:

> I began to sense that their biggest bets always seemed ultimately driven by faith in a particular cult creator, like David Fincher (“House of Cards”), Jenji Leslie Kohan (“Orange is the New Black”), Ricky Gervais (“Derek”), John Fusco (“Marco Polo”), or Mitchell Hurwitz (“Arrested Development”)... I do think that there is a sophisticated algorithm at work here—but I think his name is Ted Sarandos.

So take the case of the Nina Simone doc. It was directed by Liz Garbus (http://en.wikipedia.org/wiki/Liz_Garbus), whose previous films have been Sundance- and Oscar- fare for a few years now. So it could just be that Sarandos looked at her track record and decided that Garbus was an up-and-coming talent, and that mattered more than the specific content of whatever her next project was going to be. He was just betting that a Liz Garbus documentary was going to be great, no matter what story it told.

The lesson here isn't "Human Judgement > Algorithmic Judgement". It also isn't "Humans are good at some things and computers are good at others". It's that there isn't a good reason to make the investment in solving this particular problem algorithmically (yet).

Algorithms are great when you need scale, especially in situations where a 10% improvement in prediction accuracy can make a big improvement in the bottom line. Netflix and other studios might greenlight several new shows in a year, out of dozens that receive consideration. And the Pareto Distribution is in full effect. Most of the profits and awards come from one or two big hits. Algorithmic decision making just doesn't make a lot of sense in situations with a small sample size and uneven reward structure.

It doesn't mean that it isn't possible, though. If someone were to make the massive investment necessary to do a more thorough analysis of the content creators, the actors, the scripts and potential audiences and all of the other possible inputs then algorithms could probably do as good a job as humans, if not better. Netflix and others have only taken baby steps in this direction, working with data that is readily available and using predictive techniques that are well tested and understood. Given the nature of the problem, it doesn't make sense for them to approach it any other way at this time. But when it comes to making billions of recommendations to millions of people per day, they still rely heavily on data and algorithmic prediction. There's a time and place for everything. The time and place for algorithms in our daily lives is changing and expanding, but very slowly.

I also rather trust the algorithm to see what is a good movie and what is not. In my case the collected ratings of the experts, not the users. Current example: http://cannes-rurban.rhcloud.com/Sundance2015

There are some outliers, typical "festival hits", which only relate to festival specialists, but they are easily detectable, with a "10%" human bullshit detector. Like last years Godard at Cannes, 2012 Leos Carax, 2011 the Kaurismaki and 2010 both the Godard and the Jury winner Weerasethakul. Those outliers are even statistically detectable.

One thing is clear, you can trust the collected experts more than the juries. So I can fully confirm the story.

Rotten Tomatoes' rating, mentioned in the article as scoring THE BRONZE as 10% (though the RT now says 18%), is also a collection of professional reviewers' ratings.

The problem I've found is the algorithms for detecting taste seem to struggle greatly the further ones taste deviates from the norm. I find Netflix's recommendations to be fundamentally useless and is always reccomending things in which I have zero interest. But I've also found Rotten Tomatoes to be borderline useless as well (the second worst film I've seen in the last decade--Snowpiercer--has a very high RT score...go figure.)

Do you actually rate the things you watch on Netflix? Or go back and rate a bunch of popular movies you loved and hated? Because if you don't, you will just keep getting the same mainstream recommendations.

The title is completely misleading, seems like it's talking to the recommendation feature on Netflix, but it isn't. Everyone with common sense knows that big executive decisions aren't made solely by computers, yet the author seems amazed that big executive decisions are ultimately made by big executives. Talk about clickbait, huh?

>Even Google, the champion of algorithms, employs substantial human adjustments to make its search engines perform just right.

The author doesn't have a solid grasp on machine learning.

The 'human adjustments' provide feedback to the algorithm, which the algorithm then uses to update and improve performance. His tone implies its a bad thing to use human feedback.

Ideally, it would be a completely automated system. You don't need to understand machine learning to know that it would be better for the computer to do everything itself without human intervention

It is completely automated. It automatically incorporates human/user feedback into ranking.

Even the startingly accurate Chrome predictions?

yes

?? It doesn't require human intervention.

Simplified: Machine learning algorithm constructs page to show human.

Humans click on this or that on page.

Click data is fed into algorithm.

Algorithm uses this data decide how to show better page to humans

You are taking human input. It is still completely automated, it requires no human intervention.

How so? If the cost of making the computer able to do everything itself is greater than the cost of an equivalent system that takes human input, then you should use the human input version every time.

Monetary cost isn't the only cost.

Also, humans are prone to error and many other inefficiencies. They get sick, quit, fluctuate in performance, require attention and care... you say "cost" like it's some easily calculable number, but it's a whole boatload of intangibles that just... disappear if the computer does it.

> featuring the television star Melissa Raunch

Whoa there! It's Rauch. Kind of a Freudian slip there, considering the actress is being named in relation to a sex scene.

Here's another:

> Television studios have Nielson ratings

It's "Nielsen".

Looks like they've fixed it. But seriously don't they have editors reading this stuff before it goes live?

Dao Nguyen @Buzzfeed, "Data should not dictate your strategy," Nguyen says, "But you should understand what data tells you and also what its limits are."[0]. Heard this a few weeks back and still resonating with me.

0 -http://www.marketplace.org/topics/tech/buzzfeed-wizard-who-c...

What a weird strawman article. Did the author really think Netflix produced television shows in the same way that Siri suggests gas stations? Any blockbuster-finding algorithm (or any other component of this decision) is going to have false positives and require human intervention.

Drive such a thing thru Amazon Turk perhaps?

Might just be the "killer app" for Turks: get paid to watch movies!

MTurk has already been used to classify and tag videos.

I spent a month as Turker (as an experiment) a few years ago.

The best paying HITs were provided by a porn company to classify and tag their videos. You were given a sequence of images from the video presumably to save bandwith.

I even debated scripting a helper tool which would take a few keystrokes (for example dvda) and generate a suitable description.

This article looks at single point in the industry with regards to automation. But i believe the important point is whether the addition of data provides more efficient movie production, with less failure , and by how much.No need to automate a movie you don't produce.

In other news, Pandora is built on human input, among other things. The Music Genome Project.

Reply on news.ycombinator.com