TorchRec, a library for modern production recommendation systems | Hacker News Reader

TorchRec, a library for modern production recommendation systems

pytorch.org

94 pointstokyopanda20 comments

Just curious, what is the state of the art of recommendation systems these days?

Is this pretty much a solved problem or is there more to explore?

There's always some new neural network module that pushes the needle.

And how does it perform compared to the solution that won the Netflix prize, years ago?

Netflix gets away with mainly training on dense data from high-frequency users that binge on content. The techniques they use don't adapt very well to most other organisations.

Netflix _designs_ movies/series so that the majority of people will like them.

And so Netflix' recommendation has become largely irrelevant now that the company is pushing its own content agenda.

Their recommendation algorithm is operating under massive override by marketing rules.

There's more to explore. In particular, using groups of similar products and users to infer recommendations given only very sparse data is powerful but underexplored in production usage.

(Disclosure: I work at a a company which just expanded their product portfolio in this direction. The first few pilot customers show very good results, but there are a lot of aspects and nuances and alternative approaches we haven't had time to try yet.)

Do you feel that recommendations given to you are perfect? That more or less should answer the question.

Evaluating recommendation systems is hard because you actually require a human in the loop. Even worse, giving the recommendations alters the human behavior. Then you need to think what metric are you going to use. For training you will most probably use a proxy metric that correlates. Maybe you want to optimize different metrics and they actually need to be balanced. Then there are lot of confounding variables: maybe a better UX will improve the metrics than a better algorithm, or a change of products.

It seems that with big enough data you can improve old models with deep learning but I think recommenders are very far from similar gains to other fields (NLP or CV for example). And most companies don't have that much data.

Sure, but we don’t need to be sat next to them. A large e-commerce company can A/B test new models to improve. The success metric is revenue.

For smaller customer bases this is a tricky problem, but I’d argue that automated recommendations don’t work at a small scale anyway so manual curation is king.

If you do it right, automated recommendations work also for smaller customer bases. There's a lot of redundancy in customer and product data that can be exploited to generalise the behaviour of groups of customers over groups of products.

Doing this automatically is a huge positive ROI thing over manual curation. In fact, humans are not even that good at coming up with good recommendations. Manual re-arrangement has almost universally been an anti-relevance feature in A/B tests I've looked at.

But of course, almost nobody does the automation right.

That's the good part: putting in place a simple system will beat manual recommendations. Sadly even moderately big businesses lack sometimes automated recommendations. There are several companies that offer the service but I don't know how many clients they get. Why the disinterest? No idea, I value good recommendations as a customer but maybe it's not that relevant in the big picture.

Rec systems that tech giants use looks vastly different than what ML academics study. Industry rec systems are a constantly evolving beast. Why? Because these teams of ML engineers are constantly running A/B tests to tweak some aspect of the model. The resulting model likely doesnt fit nicely into something category of model you can describe as collaborative filtering/lambdamart/etc b/c it's some highly performant glued together mess.

Your point about academia studying different systems than industry rings quite true. Lots of the academic recommender systems are built on datasets and techniques that simply don't scale to real world data/applications.

That said, this pytorch work looks to live more in the realm of application than academia. Building and scaling large neural embeddings is pretty close to industry practice these days and this library at least claims to solve some of the challenges in doing so.

>Just curious, what is the state of the art of recommendation systems these days?

Asking your friends for recommendations. In my opinion automated systems are still not nearly as good as another person knowing you and your preferences.

It is not solved. Let me go into a bit of details: the current state of the art is using two-tower like models that create dense embeddings both for users and items. These towers are, in essence, two models that need to keep into account histories, reviews, pictures, color, material, ... . So this problem is as solved as the models that can handle these problems are.

Currently, I would expect that teams are experimenting with repurposing Attention-like architectures in some way to get better embeddings, especially from sequence like features.

There are more complex models than the one you described.

That query / item tower is cheap and cache-friendly. These tend to be used for high recall.

You can echew those performance benefits in favor of neural networks that allow for the features of the query and item to interact. These models are goaling for high precision.

Two towers are not independent of neural networks. Most two tower models I encounter are neural networks for each tower with optionally a mixing towering/layer after the two. And sometimes it's not two but even more towers depending on how your features are grouped.

I would consider two towers pretty much a necessity for large corpus retrieval as step 1 in any rec system with many items and many requests. Stage 2 or later models can be heavier and be whatever you like.

One area to explore is how to optimize a sequence of events (e.g., user journey). Reinforcement learning (RL) might be useful for that.

You'll find a detailed answer on paperswithcode, selecting the "Recommendation task":

https://paperswithcode.com/datasets?task=recommendation-syst...

Like many have already said, this is mostly an academic answer (even if some of the papers are written by the industry).

In the industrial world, the answer is a lot more subtle. Each domain has its own constraints and the best method will vary.

Also, keep in mind that in some domains like ad tech, the whole measurement process is messed up by the attribution mechanism, which attributes a sale to the last click (which clearly is a poor indication of whether the recommendation engine is doing a good job -- the industry settled on this attribution mechanism because it is easy to audit).

Recommendation will never be solved (like nuclear fusion can be) because it is about human minds liking or disliking things, and so it is pretty much "psychology-complete".

You will only see the field make steady, regular and infinite progress towards an unknown asymptote.

It’s not the primary thesis of the talk, but there’s a pretty good introduction to recommender systems: https://youtu.be/l5JqUvTdZts.

I’m a bit dated but I had a pretty significant responsibility for consumer-scale recommendation systems a few years ago, if anyone has any questions on how this stuff was done in production ~5 years ago.

Reply on news.ycombinator.com