Building the Next New York Times Recommendation Engine
open.blogs.nytimes.com
open.blogs.nytimes.com
I've been pursuing a collaborative filtering approach to product recommendation lately ('people who bought this also bought that'), but perhaps LDA would let me model our products based on their metadata ('people who bought products broadly like this also bought products broadly like that').
This means basically also finding ex. articles that are not written by NYT.
Newspapers problem is that their primarily omnibus approach to whats relevant isn't really doing the waste amount of insightful information available that exist out there.
So the whole issue IMO with all newspapers/media these days. They are building silos where none should really exist and this is one of the primary the reason why people don't consider it valuable anymore.
To your second point: how do you propose NYT do recommendation on external articles? That would require a database of external articles, for one thing...I'm not sure the results would be any better than what third-party link recommenders generate (e.g. Disqus and Taboola).
In some ways that is what Facebook is doing right now and why they are gaining more and more ground on the news-front. It's also why Twitter is kind of struggling because it's only external things leaving twitter as a protocol rather than a news service.
The whole trick IMO is to find a way to construct a whole story so i might read some stuff from NYT but then get access to more in depth on some sub subjects other places.
But this all kind of assumes that one is buying the relevance of newspapers moving forward which I am not, but thats just me.
To get sense of where I am coming from i would like to refer to some of my writing on the subject.
http://000fff.org/#/slaves-of-the-feed-this-is-not-the-realt...
and
http://000fff.org/#/how-to-think-like-facebook-and-twitter
It's about something slightly different than what you seem to imply, sorry if that was imprecise.
His idea makes sense.
Curious how they measured performance of their model, and whether they found a "best" number of topics for LDA where their model stopped getting much benefit by having more topics.
I'd imagine increased number of topics would have some interesting side effects where it would create too narrow of recommendations.
The article is awesome though good on NYT.
That said, I guess I could see a point in it maybe retaining users / subscribers if it's good enough. (I'd still appreciated it a lot more if this functionality could be turned off for users who request it though).
But after some thought on the recommendation engine, this seems more like a confirmation bias engine. Not something I'd want from a "news" source.
As the article states, it'll suggest articles about Hillary Clinton if you've read articles about her previously, but it doesn't say it'll only give you positive ones. There is a chance that it'll narrow people's interests (if you only read sports, for example) but that already happens anyway.