Comment by jacobobryant | Hacker News Reader

Comment by jacobobryant | Hacker News Reader

This is the latest of several recommender systems[1] that I've attempted to grow over the past couple years. It launched in February and there are about 800 subscribers. The algorithm uses a collaborative filtering model ("people who liked X also liked Y"), and to help with cold-start I augment the training data with content-based filtering: I use keyword extraction (tf-idf) and a pre-trained language model (fastText) to cluster the newsletters, then for each cluster I generate k "fake users" who like each newsletter in the cluster. This way, the model will gradually switch from content-based filtering to collaborative filtering as it collects user ratings.

Some of the newsletters I found on my own, but most are submitted by users (there's a "what other newsletters do you subscribe to" question after you sign up). I set up an inbound-only mail server, and I generate a unique address for each newsletter, which I use to sign up manually. I approve each issue that comes in so that we don't forward welcome emails, promotions etc. (It only takes 5 - 10 minutes a day). Before forwarding I also scrub out any links with certain keywords like "unsubscribe", "manage your preferences" and so on. It's not a perfect process but it's good enough for now.

Long-term I want to build a genaral-purpose recommender system[2]. I'm starting with newsletters because I think it'll be the easiest way to grow fast initially (which seems to have been validated so far). The short explanation is that I've designed The Sample to be extremely effective at cross-promotion[3]. (If you have a newsletter, submit it at https://sample.findka.com/submit/ and I'll send you a referral link).

[1] https://news.ycombinator.com/item?id=24921127

[2] https://jacobobryant.com/p/why-newsletters/

[3] https://jacobobryant.com/p/an-algorithm-for-driving-newslett...

Is there any mechanism to prevent you from forming your own super-echo-chamber? ML filtering only what you and your similar profiles like seems possibly have problematic outcomes, no?

Interesting, and inevitable, project, but I have my concerns.

Yes, the algorithm accounts for the exploration-vs-exploitation problem.[1] Right now it uses a simple epsilon-greedy strategy: 20% of the newsletters you receive are picked completely at random. I also use a technique I call "popularity smoothing," which limits the number of times that popular newsletters can be forwarded.

I think the concerns about ML causing echo chambers/other problems, while not completely unfounded, are overblown (perhaps due to the overall anti-big-tech sentiment). I think human behavior plus ease of sharing online is a much larger factor. I'm optimistic that ML filtering can actually help people get a much larger variety of information, which is one of my goals for this.

[1] https://en.wikipedia.org/wiki/Multi-armed_bandit

Thank you for sharing your approach, I find it very interesting. I've worked in some systems that also optimize for "serendipity"[1], which is just optimizing not only for ranking accuracy but also for adding new relevant content into the mix. (I believe you need a lot of data for this to be viable though)

> I think the concerns about ML causing echo chambers/other problems, while not completely unfounded, are overblown

I disagree that it's overblown. This is a widely discussed topic in ML research, especially around recommender systems [0]. While I do agree that ML systems have enormous potential in augmenting human capability, we should be addressing possible flaws such as the one mentioned.

I'm personally interested in how to solve biases in machine learning systems, as it impacts so much of my professional work. But I also think bias, or echo-chamber, isn't unique to ML since we see it so much in the world and institutions that surround us, but we are in a unique position to address these problems directly on the systems we create.

[0] https://arxiv.org/abs/2010.03240 [1] https://link.springer.com/article/10.1007/s11390-020-0135-9

> I disagree that it's overblown.

Fair enough. To be clear, I'm not saying ML bias isn't a significant problem; and certainly I'll continue to address it as the project grows. Based on this, it sounds like we might be in agreement mostly:

> But I also think bias, or echo-chamber, isn't unique to ML since we see it so much in the world and institutions that surround us, but we are in a unique position to address these problems directly on the systems we create.

I often run into people (usually not ML practitioners) who appear to think that ML inherently results in filter bubbles/bias (as opposed to specific ML implementations) and thus think the entire approach should be abandoned; whereas I think ML, with a proper focus on reducing bias, is one of our most promising options.

Out of curiosity, I wonder if newsletter diversity would be improved if you made suggestions by clustering the newsletters by content (similar to your other comment about seeding the network), but picking from the newsletters furthest from any cluster

Yes, I would like to do that--combine it with the rating prediction, so e.g. if two newsletters both have a predicted rating of 4 stars, prefer the one with content that's most dissimilar to the users' positively rated newsletters so far.

Instead of 20% being completely random, could you instead present some newsletters that are completely opposite to what they prefer? For example, if someone actively reads conservative political commentary, adding in some random newsletters about cars or travel doesn't improve bias (correct me if I'm wrong). It would not be better to offer some percentage of liberal political commentary? And vice versa of course. My suggestion is imperfect because I'm leaving out the middle/moderate viewpoint and touting only extremes, but I'm ultimately trying to drive toward not diversity a of topics but diversity of viewpoints on interested topics.

Something like that could be an option; see also this other comment.[1] However every time I've thought about "hey, what if we made the 20% better by...", my conclusion has been "oh, but then we might never recommend [items with some set of properties that should get recommended], and the algorithm won't be able to correct itself." Any time you try to optimize in a certain direction, you'll always introduce at least a little bias[2], and randomness at least gives a chance of breaking out of that. So something like this could be a good idea, but I wouldn't want to have it completely replace random recommendations.

[1] https://news.ycombinator.com/item?id=27669863

[2] This experience has affected the way I perceive advice from people on the internet

Hey Jacob, Why Newsletters instead of News directly? (I read [2]) i.e. Personal curated news as that would apply to everyone who consumes news and just not those who consume newsletters.

I'm specifically interested in it because I'm following a need gap - 'Get me the news I need, not the news I want' on my problem validation platform - https://needgap.com/problems/156-get-me-the-news-i-need-not-... and your solution if applied for news articles directly could address that need gap.

(FYI looks like Needgap is down right now; Cloudflare is showing me a 521 response)

There are a few reasons. The biggest one is distribution. I came up with the idea for The Sample because I was working on an essay recommender system[1] and I was trying to figure out how to incentivise people to share it. I've built a referral system[2] into The Sample, and whenever someone submits a newsletter, I send them a referral link. If they share the link, their newsletter gets forwarded more often. So far it's been promising; last I checked, about 15% of our subscribers came from referral links. As The Sample gets larger, the referral system will become more compelling (network effects). I've used a fair amount of paid advertising to get things going, but after some threshold, I think it will grow extremely fast with just the referral system.

The next reason is implementation-related. I prefer to use collaborative filtering as much as possible, falling back to content-based filtering only when there isn't enough rating data for collaborative filtering. But for collaborative filtering, you need long-lived items. If you're recommending news articles, by the time you collect much rating/interaction data, the article will be stale and you won't want to recommend it more anyway[3]. Recommending newsletters is IMO an excellent way to combine ML with human curation: humans figure out what news is good, and ML helps readers find curators who are good/relevant to them.

[1] https://essays.findka.com

[2] https://jacobobryant.com/p/an-algorithm-for-driving-newslett...

[3] Unless perhaps you're a large news site and can collect lots of interaction data quickly. An interesting thing about my situation is that I'm trying to bootstrap a recommender system from the beginning, whereas most recommender systems today were built as an add-on for another product, after it already had lots of users.

It does make business sense to deliver newsletter vs news directly as the former is B2B.

The implementation related decision sounds interesting, I need to think more about this as I wonder whether news in newsletters are anyways going to have limited lifespan. Moreover investment of time on user side for newsletter delivering news increases with each newsletter vs One newsletter delivering what I want.

Congratulations for the launch, I feel recommendation systems need innovation and so I support your endeavor.

> (FYI looks like Needgap is down right now; Cloudflare is showing me a 521 response)

Sorry for that, It's been resolved now. You're welcomed to join the discussion on 'Get me the news I need, not the news I want' by explaining what Sample does to address that problem.

> Moreover investment of time on user side for newsletter delivering news increases with each newsletter vs One newsletter delivering what I want.

You might be interested in this article[1] and the discussion of subscribed vs. filtered sources. It'd be nice to have a convenient way to assign different priorities to newsletters/content sources. So for a few, you'd get every issue, and for the rest, they'd be combined together in some way.

> You're welcomed to join the discussion on 'Get me the news I need, not the news I want' by explaining what Sample does to address that problem.

I'm not sure I have much insight about needs vs. wants, but you might be interested in The Factual.[2]

[1] https://jacobobryant.com/p/the-pros-and-cons-of-algorithmic/

[2] https://thefactual.com/

In the words of the bayesian investor[1],

> Do you want to focus your attention on problems that are most likely to hurt you?

That's the difference between want vs need. I may want to read all the news posted on HN, But only a fraction of it might be what I need in terms of what I need for betterment of my well-being, career etc.

[1] http://www.bayesianinvestor.com/blog/index.php/2021/06/13/av...

Reply on news.ycombinator.com