Online Monoculture and the End of the Niche

"The set of 48 customers is divided into equal-sized communities, with members chosen at random so they may not be close in taste.

The recommender function chooses an item by looking at what customers in the same community have chosen. It recommends the one most popular among others in the community."

So first we randomly assign you to a group and then suggest items that has already been marked as popular. With an algorithm like that it is no wonder why he ended up with the data he did. This is the model where we all go and buy from the amazon top sellers list, not because they are good but because everyone else is buying it. There is zero surprise that showing this list to users would cause the items on the list to go up. This is also about the worst recommendation algorithm I could think of. His algorithm is the one used before the internet by the general population. You don't listen to every band, but see what your friends are listening too. You buy the same fridge as your friend because they recommended it. You did things usually because other people had, not because they were good. So according to his data what was happening before the internet should never have happened.

A good recommendation algorithm will present me with choices that I actually want to see. The key is that the choices presented me are only my choices and not choices that everyone sees. So as time goes by the odds that I will see the Nova episode on Rats from a few weeks ago will grow higher (really good one btw). But I don't think tivo will suggest that to all tivo's.

In a real recommendation algorithm as time goes by good items will get viewed more and bad items will not and the spread will grow larger. This is exactly what Netflix people talk about when they say they have been shown all sorts of great movies they never would have tried on their own. Netflix hates, absolutely hates monoculture because it means they have to buy 1 zillion of the blockbuster dvd from the summer. Which is part of the reason they do recommendations and sponsor its research. If they can find you something better to watch it means they can spread the que of users out to not be a monoculture. That is what recommendation algorithms do, they create niche's.

"The result is what's important here, rather than the particular algorithm used to generate this instance of it."

Ah, the "computer modeler's" rallying cry. Something that has always worried me about performing "experiments" on computer models: software has bugs, and the investigators building and using them have preconceptions. The two interact to collectively bias results towards fitting the modelers' preconceptions: you fix bugs until the output "looks plausible" and fits some control data, and then stop.

Shorter version: if you're not in a box, you're probably going to see more things, and if others aren't in a box, they're probably going to see more things, too, but the odds that those things will overlap goes up.

Then the things that were in the box that aren't getting the same amount of attention paid to them because people have more choice, those things get their feelings hurt, or something.

Add dubious computer model and some inscrutable graphs.