Comment by jillesvangurp | Hacker News Reader

Comment by jillesvangurp | Hacker News Reader

I've been toying with LLM based agents for staying on top of feeds lately. Staying on top of the fire hose was always the challenge with RSS readers, a lot of interesting low frequency stuff from people you follow gets buried under the non stop feed of stuff from high volume news that is actively trying to drown everyone out with many updates per day. Back in the day I solved it by simply not using RSS feeds for high volume stuff. But I was following a lot of interesting people. I miss that.

This is what lead to algorithm based filtering. Hacker News uses a simplistic algorithm but it is definitely using one and it works well enough. It's why I come here. We all collectively vote things up and what remains is nominally interesting enough to skim from the front page. With a bit of editorializing.

Social networks tried to game the algorithms for ad revenue. Which is why they are a lot less popular these days. Sites like Medium, Substack, Tumblr, etc. took over from simple blogs and immediately started raising walled gardens around them to become discovery platforms, have recommendations, etc.

But at least they support RSS. A lot of websites still do. If you run any kind of website publishing regular news or article content and you don't support a feed, you are being an idiot. It's easy, doesn't really cost anything, and you might actually get people using your feed once in a while. Your site might actually have one without you realizing. Most news papers have feeds. They are everywhere. The main issue isn't finding them but sifting through them. It always was.

With agent based approaches, you control the algorithm. That wasn't possible in the past. LLMs can summarize, aggregate, categorize, group, filter, etc.

> a lot of interesting low frequency stuff from people you follow gets buried under the non stop feed of stuff from high volume news that is actively trying to drown everyone out with many updates per day.

Every major RSS reader supports folders. Your problem is that you engage with RSS as if it were a social media feed, with it's single monolithic reverse-chronological feed.

Just don't do that. Stick all the high volume news feeds in a folder, and you can skim read the headlines & hit "mark all as read" once you're done or for whatever other reason don't want to look at the news anymore.

Stick the low volume things you care about in their own folder, and those will remain unread, in their own ordering for you to read at your own leasure.

Even for sites that don't offer granular feeds, every major feed reader offers filtering options, a lot of them offer fairly complex regex filtering.

> This is what lead to algorithm based filtering.

Feed aggregators (and most social media) exist because of discoverability, finding new stuff from new people you hadn't heard about before.

> With agent based approaches, you control the algorithm. That wasn't possible in the past. LLMs can summarize, aggregate, categorize, group, filter, etc.

You'd be spending tens of dollars of compute on something that every major RSS client was doing back in 2006 with the equivalent of less than a single penny worth of current day compute.

> Even for sites that don't offer granular feeds, every major feed reader offers filtering options, a lot of them offer fairly complex regex filtering.

This is true to some degree, but regex filtering or really any strictly logical filtering is often too coarse or too fragile to work well in practice for all but the most dedicated RSS gardeners.

What I’d really like to see is some sort of fuzzier logic that gives the user a more semantic interface to filtering and/or ranking feed items.

Take, for example, the feed for a medium-sized newspaper where you want to filter for/prioritize articles about your local area and particular topics of interest. Those news feeds are often very high volume and don’t make good use of tags or other metadata that can be consumed by strictly logical filtering systems. So a fragile, badly-behaving filtered list is what you’re likely to get. Whereas a fuzzier, more semantic interface (local LLMs?) would be far more reliable and easier to use.

Just as a suggestion, but I've been thinking in terms of clustering. If I can figure out that five different posts from five different feeds are about one story, I can collapse them into a single UI element.

I did exactly this with Cohere language embeddings

https://aws.amazon.com/blogs/machine-learning/use-language-e...

Thanks! Something to dig into.

Are there existing good implementations of that? Does Feedly have that?

I'm building Subweb.net (not ready yet, it's just a few test feeds without the LLM pipeline turned on yet) to LLM-tag RSS feed items with topic, relevance/interest, location, and translations, and present them as feeds. I'm thinking I could maybe let users specify their preferred custom prompts and ranking params or similar, though the standard prompt is already fine.

I think the open web needs to come back, but in a fair way for everyone, giving readers control over their feeds while also sending traffic and comments back to the original sources. Not quite sure how to do that yet.

I curate my subscriptions pretty heavily. I scan the feed and triage to my linkding instance. I have a headless app that fetches unread items, uses a voice model from Hugging Face to audio using piper and then adds them to audiobookshelf as a "podcast". I catch up on things when I'm on a walk or doing the dishes.

I built clawfeeds for that (forum own use). Just tell Claude, check out https://agent.clawfeeds.com/howtouse.md

Reply on news.ycombinator.com