Categories of deep recommendation systems
jameskle.com
jameskle.com
I still find that human based recommendations for things like music and movies consistently beat the algorithm at predicting what I'll like vs what I won't like just about every time.
IMHO it will be many years before these recommendation engines consistently pick media I like to watch or listen to.
They are still far inferior to human based recommendations.
Guessing what someone will want to watch at any given moment is probably a harder problem than getting a car to drive autonomously.
EDIT (source): I work on a mostly human powered recommendation engine/expert system @ lazyday.tv. It does not use ML to make recommendations. It takes some creative approaches to solving the movie recommendation problem using an expert system and optimizing search results.
https://opensourceconnections.com/blog/2016/08/21/recommenda...
My YouTube subscriptions are all engineering/programming/space/guns/machining/electronics. I even see the same commenters in many of them. There is clearly a set of people with these interests that could be used to cross-pollinate interesting content. Yet my recommendations are all pop culture and sports.
Spotify is the same way but i blame my kids for polluting their model of my interests,
I’ve given up on Netflix. I can’t even find stuff i want to watch on there any more.
By comparison the known movie/tv universe of global movies and tv selections ever made is upwards of 500K.
Something I've been wondering about recently is how infrastructure costs factor into this. Obviously videos get served from a local CDN node when possible, but I'm guessing sometimes the file you're requesting is uncommon enough that the local CDN node doesn't have it. Maybe this isn't the case with Netflix, but it's probably the case with youtube, just because they have that much more content. Adding more storage to the CDNs would cost money, and cache misses on the CDN also cost money, so does youtube have a financial incentive to bias their recommendations towards videos already cached near you?
An unrelated problem with movie recs is that the kind of movies & tv episodes being made today have homogenized because of international distribution issues & things like metoo. Your villains can’t be clearcut russians & chinese anymore, we sell to those countries. So it has to be some amorphous bad person with ambiguous intentions, which doesn’t work so well for james bond type stuff. Many of the older shows I love, I watch them on youtube & the comments there are mostly along the lines of “this would never ever get made in 2019. this is so bigoted. this is so mean spirited. this is openly sexist racist” etc etc. These are shows from 70s-80s, so its not like we had some genetic mutation and became a different species. We are still the same people, but those kinds of movies & shows aren’t happening anymore.
I don't see how collaborative filtering can not work. Especially in the case of modern media providers where they have data about whether you actually finished watching something, how many videos of a series you watched etc.
I'm not familiar with what the best approaches for the cold start problem of getting that first (few) interaction(s) are, but in the case of Netflix it's easily handled by a new on Netflix section.
A statisticians would say that you are making "heroic assumptions" about both the taste of viewers, and the dimensions of the product, especially given that the products you have are actually quite sparse. Heck, there's even interesting literature in empirical economics (called empirical industrial organization) trying to estimate prices/demand/costs (which is, essentially the same issue as Netflix trying to estimate what you like) and dealing with the inherent curse of dimensionality. This becomes really obvious if you are actually write down explicitly what you are doing.
Of course ML engineers are engineers and not really interested in causal inference, so Netflix wants me to watch 12 Hitler documentaries in a row and Amazon recommends that I better buy that second fridge right after the first.
Where _does_ collaborative filtering actually work?
That said, the competition is hardly any better. Most use the same collaborative filtering from the Netflix Prize (Koren et al), which by design never recommends anything new. Most new "big data, machine learning" personalized recommendations just give you most popular plus some random noise. They perform well on some very artificial evaluation functions, but rarely A/B tested on actual users. When they are tested, they barely beat naive models like IMDB top 250.
I mean they have entire divisions of engineers dedicated to this so I assume they will disagree with you. So I am wondering if you have evidence that all those people are not doing anything to earn their pay
Personally I think it's amazing that even basic recommendation algorithms can instantly select 10 mostly relevant items from corpora of millions of irrelevant items. I think people are too used to receiving excellent recommendations and do not realise what receiving bad recommendations would be like. Hybrid systems such as Spotify's are really amazing when you think about it.
But there is no ML compiling a profile of every genre/title you've ever expressed interest in and then using that profile to guess what you want to see next like Netflix et al.
It is certainly faster to build an expert system that makes a lot of assumptions and iterate from there. There are plenty of use cases for ML and I hope to integrate it more into app some day to help automate repetitive tasks.
Am I correct in assuming Lazyday has adopted a rule-based approach, using information about the movies (e.g. release date, genre, director, cast, etc.) and users (e.g. age, sex, nationality, current mood, etc.) that collaborative filtering systems don't use, and that you have evaluated the system against a collaborative filtering system which uses the ratings your users provided?
I developed an online collaborative filtering movie recommendation system in the mid 1990s ( http://www.fmjlang.co.uk/morse/MORSE.html ), and analysed its performance, for my MSc project. The only information it used was how each user rated the movies they had seen. It used no other information about the movies or the users themselves.
A surprising result I found was that using the ratings of the person whose tastes most closely matches your own to predict your ratings was very inaccurate. (Note that this is not necessarily the same as someone recommending you a movie, as they might know and take into account your tastes, and even if they didn't, they could still improve their recommendations simply by recommending the most popular movies, rather than the movies they liked.) Expanding from a single nearest neighbour to the average of the N nearest neighbours (for N = 25) gave quite good results. It seems that you're more similar to an average of all your friends than your closest friend. Using the algorithm I developed (described in detail in the paper) resulted in a further big improvement, which was unsurpassed until the Netflix prize. My data set was significantly smaller, though.
Yes Lazyday is a rules based search engine. It makes best guesses based on what's currently popuplar, the movie's meta data and user search preferences and history. We have not evaluated it against a CF system in an A/B setup. Of course we have years of user frustration with Netflix / Prime recommends that has taught us a few valuable lessons about how to solve the problem of what to watch next.
And yes it seems intuitive to assume that two people with the same tastes will like the same titles. The problem is, I might not be in the mood for movie A even though my virtual twin is, sometimes I would be in the mood for movie A if I had company that was also in the mood for movie A. So many edge cases when a recommendation engine is trying to guess what you want to watch next.
Using the averages of your friends is very interesting. The issue would be building your dataset. Attaching it to a network like FB or building it yourself which would take even more time.
For example, I watch a lot of cooking shows. I just scrolled to the bottom of the recommendations on my tablet and they only suggest two—-and one of those I’ve watched every episode!
They repeatedly fail to tell me about new series of shows I’ve watched, or really obvious connections like “Breakfast, Lunch & Dinner” as I’ve watched “Ugly Delicious”.
The problem seems less of a ML one to me and more of a (lack of) metadata one.
Edit: I plugged “Ugly Delicious” into lazyday.tv and the “Similar Titles” are very random.
These systems almost never work in production. The publications are usually very deceptive.
I clicked "Start radio" and the first song was "light", of course. Then all the next 10 songs were by Idyllic only, no other artists. After that half the songs were Idyllic, the other half were....songs from my "Thumbs Up" playlist (any song I had previously liked).
I repeated the process several times thinking there was a bug mixing in my thumbs-up playlist.
Eventually I realized I'm the first person to ever listen to this song on Google Play Music and it used only me as the first bayesian filter - so I just got my own music suggested back to me.
Instead, this was an index of current literature from pretty much all deep learning methods that have been applied to recommender systems. It's more useful to a lay person than a researcher.
A few years ago I made a recommender based on critics’ inferences in reviews. For the right kind of film (indie or festival circuit stuff and the like) it yielded quite interesting results.
Probably too long to fit, op did a decent job of compressing imo.
The original is:
RECOMMENDATION SYSTEM SERIES PART 2: THE 10 CATEGORIES OF DEEP RECOMMENDATION SYSTEMS THAT ACADEMIC RESEARCHERS SHOULD PAY ATTENTION TO