Combine multiple RSS feeds into a single feed, as a service | Hacker News Reader

Combine multiple RSS feeds into a single feed, as a service

github.com

61 points8organicbits32 comments

I cannot see any use I'd have for that. All the RSS readers I've used in my life have supported reading from multiple feeds. Could someone enlighten me?

I guess this moves complexity away from the client and enables a dead simple setup (maybe for non-technical friends and family?) with just a single big feed, like they already know from TikTok and the likes.

Maybe this could be used to offer aggregated feeds as a service. It could be further refined to filter content based on criteria either provided by the user or predefined by the service. An example could be something like an infosec feed with curated sources and maybe an optional refinement for content related to vulnerabilities.

This could also allow users to develop/fork the aggregated feeds and share them with a community.

This in combination with the <source> element from an item is a killer combo.

If you use rss as a middleware protocol, it saves you in client configuration. You deal with the updates and changes on the sources, for your client app it is just one never changing feed.

Tbh, I run specialized vertical services aggregating 100's of sources (rss as well as other), and the combination towards a single feed is trivial.

I am equally baffled.

It's a pet peeve of mine seeing software projects that describe what they do, but without providing even a single use case. Or even what motivated the developer.

Why does this thing exist? There is probably a reason and then I could learn something.

A bunch of friends and I once toyed around with the idea of replacing microblogging through ActivityPub with one RSS feed per user. To then display them in one combined feed, I just added all of them to a category in my RSS reader, the way I already did with topics before. One of my friends' RSS reader didn’t have that option (grouping by category yes but no combined feed) so they instead combined the feeds into a single RSS feed and added that to their RSS reader. In the end it was kinda finicky so we went back to ActivityPub.

I'm not sure if this is the functionality you are looking for, but Mastodon has built in RSS feeds per user. You just to add .rss to the user link. So, for example, if you had an account on mastodon.social, thered be an RSS feed of your posts at mastodon.social/@sussmannbaka.rss

Yeah, there are also third-party services that provide RSS feeds for additional Mastodon data like tags as well. e.g. https://openrss.org/mastodon.social/tags/assange

I think koreader[0] has an RSS feature where you can fill in some RSS feeds and it converts feed items to epubs. It's nice, but I'm using a different reader on my laptop, so then syncing (of which feeds you follow) becomes a thing to do.

Instead of following a bunch of them and having to sync them I guess you could just follow the combined one and be done with it. If you have multiple feed readers, each with their own way to input feeds, I think something like rsscombine might be an easy solution.

[0] https://koreader.rocks/

Check out planets[1] as one use case. Projects like Debian, Mozilla, Python, and many more aggregate feeds into a planet to promote the blogs of members or other people who work in their space.

Being able to subscribe to "Debian blogs" in one step is helpful versus discovering each blogger individually.

There's a big list here [2], I think two use the linked code.

[1] https://indieweb.org/planet

[2] https://github.com/ArchiveTeam/urls-sources/blob/master/4320...

I saw this on HN and though “great I was just about to write this but now I don’t need to” - I have a legacy blog, a link blog and another thing I update which all produce RSS and I want to create one big feed of stuff I write, regardless of which platform I chose to put it on. Seems like I can use this for that.

But almost every RSS reader has this feature.

I use FreshRSS and you just put all the feeds in one category and then you can click on that category to show all the feeds together.

The reader has to add every RSS feed. What if I as the producer of those feeds want to ship a combined feed so they don't have to do that extra work? Or what if I'm a curator of RSS feeds and want to release one combined "low-tech computer news" feed sourced from 100 smaller low-tech computer news feeds?

This is already solved with dynamic opml, which freshrss implements

I’ve wondered if this existed, to simplify writing a system that parses and processes RSS.

Frankly I’d go a bit further and spit out events, so people can code “give me the next entry” without having to mess with polling lots of rss endpoints. Just tell me when there’s a new entry.

Or turn it into a web hook so you just code the parsing, and delivery is outsourced.

Polling for changes is a terrible design. The source has to serve data on repeat to every reader that is set up to check once every week/day/hour/minute.

At the very least have a distributed cache that is set to return only the entries created after generally agreed timestamps. (Please tell me that exists now, or why it’s a bad idea.)

> (Please tell me that exists now, or why it’s a bad idea.)

Regarding the only-after semantics:

- HTTP has "Delta encoding in HTTP": https://www.rfc-editor.org/rfc/rfc3229 (2002),

- usage for feeds proposed in 2004: https://bobwyman.typepad.com/main/2004/09/using_rfc3229_w.ht...,

- some (not so) recent discussion (2017): https://www.ctrl.blog/entry/feed-delta-updates.html + https://news.ycombinator.com/item?id=26091489 (zero comments).

Adoption (or lack there of) is a different story. I guess the main issue is that it puts more work on the publisher side, in particular requires dynamic aspect when generating the response.

I used yahoo pipes for this back in the day. I wanted to combine multiple feeds into a single date sort. So it gave me all the issues on like 20 repos from oldest to newest.

This was much faster than clicking on 20 different feeds to see which ones are oldest or most recent.

It will work well with RSS Proxy https://github.com/damoeb/rss-proxy which turns any website into an RSS feed.

Hah, I've recently had to solve the exact opposite problem - fetching a bunch of content from a single source, and splitting it into multiple feeds.

Amen! The Android Developers blog (https://android-developers.googleblog.com, available as RSS at https://android-developers.googleblog.com/atom.xml) comes to mind - It includes updates on new compliance requirements to keep an app listed on Google Play, which I care about, mixed with a whole lot of crap I don't care about, like Google's latest shiny thing like Jetpack Compose, fluff pieces and marketing speak.

How did you solve this problem?

Hand-written code :)

I don't split an existing RSS feed, but I scrape a website, and store the posts in a database, with a column for the user who posted it.

Then I generate an OPML file with URLs like http://localhost:1234?user=... per user, and spin up a web server that fetches only that user's feed given the query parameter. Import the OPML into the news reader, and bingo bango.

For my personal site I create RSS feeds for all the main posts, but also an RSS feed for each tag. If someone only wants to read my posts about TypeScript they can just follow the TypeScript tag instead of the main feed.

I needed something like this once but I forget why. Now I would probably just use val.town to build it instead of spinning up my own go server.

I miss Yahoo Pipes so much.

I use Node-RED for largely the same things these days, including aggregating and summarizing feeds with gpt35-turbo.

Hey, that's interesting to me. Could you share a bit more about that? Do you replace the feeds with one summary item?

No, I collect all the items, save them to SQLite, and then batch them to Azure Open AI, generating a summary feed where each item is a set of summaries grouped by topic. One feed turns into 2-3 items a day in 8 hour windows.

I'm not claiming it's equivalent to Yahoo Pipes, but my https://pipes.digital does have a combine block ;)

There is one footgun with many tools like this. If you add a new source to an existing combined feed all of the items in that feed will immediately appear as new items. What you typically want is for new items for the source feed to show up in the combined feed, not all exiting items at once.

Listen Notes lets you do this for podcasts:

https://www.listennotes.com/blog/why-podcasts-are-my-new-wik...

FreshRSS does this.

Reply on news.ycombinator.com