Steam has exactly the same problem, although ar least they‘re trying to solve it in various ways.
Amazon is notorious for trying to sell you the same kind of product after you've already purchased it (e.g. I bought a ladder and now all of my recommendations are for more ladders). How hard can it be to uncover which items are more likely to be one-time buys and recommend the products people buy with those items instead (ex. show me painting supplies when I buy a ladder)?
It is hard. Dynamical systems.
My hunch is that they aren't trying to improve those algorithms very hard. But it would be fascinating if there is something about human nature that I am simply not seeing here.
First your information goes into a system. That system is used to reason about how you buy things, and how someone similar to you buys things. This happens for every individual in the system.
This eventually yields a cohesive model of behavior that can be used for probabilistic inferencing which effectively is establishing real time dynamic 'types' of both 'items being sold' and 'consumers making purchases'.
Because it's so heavily generalized to a probabilistic/stochastic model, things that shouldn't get connected can get connected. You know the age old adage, correlation is not causation? That's exactly the problem with a model that is based entirely on relational linking of items with little to no oversight in ways to explain how it's actually functioning. I want to use the word self-referential because that's very much what these systems seem to be doing, it's like it literally loops in making one too many inferences about it's predictive space. I'm not saying this is precisely why it returns your choices back to you.
But given that there's been some research (that I honestly haven't kept up with very much) that comes down to rigorously defining a data 'context' (for something like an RNN) and then rigorously searching the space for all data of what is a 'perfect match' for anything, all of this hyper perfect matching is bound to return you back to yourself in funny, ridiculous ways.
And yes, I realize the many layers of irony for those people out there who would like to point it out. I can try to explain it and I'm making all the same mistakes it does. Trying to reason about this stuff is NOT easy because it makes your brain feel broken. And I don't build RNNs or anything like that, I've done a very small amount of research on recommendation engines when it was early days, I've taken engineering classes that study probability and stochastic processes which are used to model systems where the 'types' of variables are not always easily distinguishable from one another, but they must be organized and ordered into a system that can be reasoned about rigorously. It's like building circuits for people's minds and behaviors when they in their weakest decision making spaces (for lots of people). It's sometimes screwed up but you know, you know what you like, right haha, lol. Just blame Bezos he's the one with all the money
For content, I also find that user curated lists can be far better than automated systems. They give me an upfront sense of the curator’s tastes and define niches that algorithms would never recognize
Not to mention, they probably have several hundred times the amount of data to use for recommendations since a user changes song around every three minutes. Multiply that by the number of users and average number of songs someone listens to in a session... that's going to be a big number.
Also, music choice is mostly preferential. Some preference is used when buying products but most of the time I feel like there's a need which influences the product purchase and this factors more heavily into the decision.
Netflix I would say is a good counter-example of a content provider which is based more on preference but again, the content takes longer to consume so there's likely less data than Spotify. Not to mention, it seems they've been giving more recommendation preference to their originals lately.
I bought a LaserJet printer. I went into my "Recommended for me" and the first four pages were probably 30% other printers, and 30% printer supplies (for my printers and others).
How is this still a problem?
This is the fallacy of machine learning: these systems aren't smart, they're incredibly dumb.
They also know which printer it was, when it was bought, and presumably they could know how long printers last, if not how long that specific printer lasts.
The first is certainly enough not to recommend supplies for a different printer.
> This is the fallacy of machine learning: these systems aren't smart, they're incredibly dumb.
Here, I agree. They're generally no smarter than the people programming them.. reminiscent of any computer system.
I'm not sure what you mean by "correctly," but in general, if you're talking about automation and recommendation engines, my answer is "no." Of course, I don't know the goals of the companies implementing them and whether my sense of "correct" recommendations matches theirs. I don't think any of us do.
Perhaps the answer is simple: Amazon is somehow making money from the example I cited - more money than showing the consumer relevant extras. Or they've done research to show there isn't much opportunity for more income by improving the algorithm for this.
I personally haven't had any issues finding the books I want, and I find Amazon's suggestion algorithms helpful precisely because it works as you describe. Almost all of my book purchases are made because somebody's recommended a specific book or author to me, or because I'm interested in a specific topic or author and searched for them. In both cases, I'm fine with getting generic suggestions based on the author or topic.