A List of Recommender Systems and Resources

That what we like can't be boiled down to an optimization function over a sparse matrix.

The trick is to find stuff that cuts across the space in some sense—stuff that's out of your current wheelhouse, but still has features that you would like. (E.g. I like the music of both David Rawlings and Mark Ribot, even though they are in wildly different genres, because they both have a certain ear for melody and relationship with dissonance that I like.) I don't think anyone is doing that well yet.

1. Evaluating recommender systems is REALLY hard. When it comes to evaluation, recommender systems are very different from other machine learning tasks:

Let's assume the classic methodology of collecting a dataset of events, splitting it at a certain point in time to get a training and test set, and checking the precision and recall

- If your recommender system works as expected, you will influence what your users do in the future, so your test set is probably not going to represent what your users would actually do.

- Having a precision and recall of 1 is actually bad. That means your recommender system was perfect, but also useless (you only recommended stuff that the user was already going to pick anyway).

One way to address this is to just use A/B tests and try to optimize some business metric (e.g. number of purchases). This is usually "good enough", but this will make your recommender focus on sales, not on user satisfaction.

There are also some other metrics that can be used[1], but there are so many of them and some are not very practical to implement, so I guess that everyone just goes with the A/B testing approach.

2. Recommendations require an explanation

A lot of recommender algorithms are black boxes. Sure, you can write "this was recommended based on users similar to you" when you use a collaborative filtering algorithm, but that doesn't help much.

Recommendations without an explanation are not that useful. When a friend recommends you a movie, he'll also tell you "why". Otherwise, it's really hard to make users trust your recommendations (especially if the recommender system recommends something outside of the user's "comfort zone")

I've noticed that a lot of sites have improved on this front though, and I do enjoy those recommendations a lot more.

[1]: Herlocker et al., "Evaluating collaborative filtering recommender systems" http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97....

This actually makes recommender systems things that train humans, that render them more predictable, by grouping them into categories base on 'latent variables'.

It makes everyone more boring by reducing diversity instead of encouraging it.

You can easily see this in your youtube recommendations.

Jaron Larnier calls these kinds of algorithms manipulative - they train people to optimize engagement metrics, rather than sometime qualitative like quality.

https://livestream.com/internetsociety2/cybercon/videos/1662...

And then because the ratings come out so incredibly divorced from a user's taste, netflix naturally gives up on the rating system, and makes its judgement based on your interactions with the video: how much of a show you watched, how quickly you binged, how quickly you ditched it, etc.

But your viewing habits are not indicative of your taste, and is often quite divorced from it. Everyone has their guilty pleasures; those trash shows you watch because its mindless entertainment. The vast majority of the stuff you watch is not what you wished you watched. That's not the shit you want to be recommended, if you care about being recommended things (otherwise, high popularity is more than sufficient).

But thats because netflix does not have the goal of showing you good movies and shows, things you can watch and actually remember three days later. They have the goal of mindless entertainment (they need your time, not your brain), and the recommender system succeeds at that task. But its difficult to see that goal and think "yeah, thats the personal recommendation I always wanted". It's not what your friend would recommend to you; it's the absolute opposite.

Part of the problem is that a proper catalogue of your ratings for various shows/movies is likely to be a very good indicator of taste, but afaik, the western media has no culture of it, no decent tooling, and no decent systems for it. I know of no decent system to aggregate and curate my movie collection, adding, maintaining, and noting a simplified opinion of it (a rating), for my personal use, beyond the physical/file system storage of it, and half-assed custom-built systems for it (eg text file). Which also implies there's no public dataset of it from which to build a proper recommendation system from.

However, I know at least the anime community offers us a possibility of redemption. MyAnimeList (MAL) and co. do offer a decent system of it, and the community itself lends itself towards the cataloguing task[1]; in my own usage of MAL, I consistently go back through and update my ratings to reflect my current taste, and more importantly, the difference between an old 10-rating and a new one, when I see something that beats out by previous by a long shot, shifting the whole rating. And I'm willing to do the work because maintaining the list is relatively simple, and it offers me great utility; both things you won't find by amazon or netflix.

And I expect others use the site in the same fashion. I think if you really want a decent recommender system, this is the kind of dataset you want. Something that actually happens to reflect a user's taste, because the users themselves are incentivized into maintaining a simplified snapshot of their taste, because they can actually derive value back out of it.

I've had intent for some time now to do that, but never moved beyond the scraping stage (some 40GB of MAL lists sitting on an aws db for almost a year now). Also I'm pretty damn sure with such a dataset, you can get really far with a few simple intuitions (picked up off random papers and my own thinking):

1) User definitions of ratings is mostly arbitrary, but people trivially agree on at least this much: in a 0-10 scale, anything below 5 is disliked, and anything above is liked. 0 is trash, and 10 is good. Normalize on it.

2) Anime communities have a notion of "dropping" shows. Watch 3 episodes, and if you don't like it, move on. This is probably true in all media. These extremely disliked shows are likely not recorded (though MAL has an item for recording dropped), so ratings are inherently skewed positively.

3) The average taste is bland. You have to cluster before you can do anything.

4) No one wants the average. They want the best, in their own little zones. You don't want the centroid, you want the edges of the cluster.

5) No one wants only the shit they already know, and its easier to steal from someone close to the target (in taste) than to generate a personal recommendation. Find nearby neighbors in taste, then find the anime they like, but doesn't belong to the cluster. (this kind of find is by far the most valuable; thats when a recommender system actually offers something you probably wouldn't have found on your own)

6) People are naturally more willing to accept deficiencies in someone else's list, than a personally generated one. Instead of only generating a new list, give a list of the user's neighbor-lists as well.

7) Point 5 is good for the community: If their taste is that similar, shouldn't they be friends? Or enemies.

8) Rarity is far more important than rating at the extremes. If two users even list the same rare anime, they probably have a lot in common (they managed to find it, meaning they're in similar communities, which implies similar tastes, even if this specific one differed). The most popular anime is worthless: everyone has an opinion on it, and most likely not a useful one. (TF-IDF)

One day I'll do something with this. Maybe.

But I'm pretty damn sure you can get really far with very little. But the ratings systems are generally built poorly, and so the data they generate is worthless, which leads to a worthless recommender system. And to work around the poorly built data, they convolute it with all sorts of fancy ideas to get around it, but never actually fix the data generation issue.

Notably, one thing I've never found a satisfactory answer for is how to evaluate success/improvement. The best metric I've heard yet is split an animelist in half (or more like 70/30), use one half to predict the other half. Eh.

[1] My favorite description of this tendency is Otaku: Database Animals

[2] Anime Booru's are basically an amazon turk-built ML-labaled dataset; community constructed long before ML was in any way notable. They are accidentally ideal for ML. I'm pretty sure the west accomplishes something similar with porn datasets, but to a much weaker degree.

Netflix < MovieLens < Criticker < Coollector Movie Database