Homemade RSS aggregator followup

You should not be afraid of open sourcing code you can't support or are not happy with :)

Here's my most popular Github repository. I never use it and it's entirely supported by its users, I just keep an eye on it:

https://github.com/jleclanche/django-push-notifications

And here is some of my worst production code. Fully undocumented and with my first Python code ever in it, never rewritten/cleaned up:

https://github.com/jleclanche/pywow

There, I just showed you terrible code and unmaintained crap ;) Don't be ashamed!

..was just thinking out loud :-)

But one advantage is that this is immediately viewable on any device, because otherwise you would need a WARC viewer, which are not so easy to setup.

RSS tells you what's actually a new page. Usually, if a site wants to be Google-friendly, it'll have a page for every new content item—so, if a site publishes an RSS feed, that's usually enough to "chunk" their content by time.

Heuristics to decide what's new on a site are much, much harder to code than heuristics to extract content when you're explicitly told what's new. There was a service called Dapper, who tried the "extract content from a CSS-selector-specified zone of a page when content changes" approach... and it didn't go well. Yahoo Pipes had similar aspirations; again, got shuttered.

There are always sites that are really bad internet citizens. Some sites just change their front page to update, without creating a canonical archival URL—so then, if they do publish an RSS feed, every item is just a link back to said front page. There's not much you can do in these cases beyond just crowdsourcing "parsers for hundreds of sites" (which does happen—webcomics being a frequent and horrifying example.)

But there is a pretty good alternative, I think: if you can scrape the site itself as a whole at regular intervals, with fine enough granularity that any two contiguous scrapes will detect either 0 or 1 change, then you can probably convert that into a useful RSS feed without trying to figure out from "what changed" from DOM deltas. Basically, shove each scrape into the working index of a git repo, commit, and generate an RSS feed of the diffs. (Wikipedia can give you this in some form; I think gwern.net has an RSS feed that also basically works this way.) And, if the site has even the most minimal source RSS feed, you don't need the "regular intervals" approach: you can use the source RSS feed to provide "cueing" information (i.e. timestamps of when the site has changed) for your scrapes.

Which is all to say, RSS is a very "utilitarian" format. You don't need to rely on all of this processing happening on the server side, but rather can just take one RSS feed, and then use whatever bits of it you want to generate another RSS feed, and then someone can consume that RSS feed and write a heuristic to generate cluster and combine the feed items from it into higher-level summaries, etc.

This sort of thing really shows the "theme" of RSS, to me. RSS wasn't really designed or intended for direct consumption by readers, but rather to make it easy to have something on your site (even your statically-generated site, or your crap one-off PHP blog) that's easy for other services to consume and turn into stuff. It's "Really Simple Syndication": it's not the whole process of getting stuff to clients, it's just the lowest barrier-to-entry to get the supply-side of the equation involved, so that well-made delivery infrastructure can take over from there.

There were never meant to be "RSS reader" software, really, or even e.g. consume-side RSS-to-email gateways. Instead, RSS was meant to be lowest-common-denominator format for other supplier-side services to consume. Instead of writing "lifecycle emails", for example, you were meant to just have your web app generate a lifecycle notification stream RSS feed for each user—and then subscribe an emailing service to it.

We've replaced this behavior, for the most part, with webhooks—having our web-apps actually reach out using a REST client and prod other services' APIs. But RSS (with one consumer—the hooked service) is much, much cheaper than a REST client, from all perspectives: your web-app can likely already generate XML, it likely already has a list of changes and "view rendering" capabilities, etc. Webhooks are for the 1% of apps that can both 1. run server-side, 2. reach-out from their sandboxes to poke something elsewhere on the web, and 3. have a public URL where they can be poked back at. An RSS-based pub-sub event system, meanwhile, will work even if the event-generator is on your local machine: as long as you proxy over the RSS feed URL itself, you don't need to worry about machines trying to figure out whether the site that prodded them with a webhook request is coming from the right IP for that client or not, etc. They just get to blindly consume a URL, like any other piece of idiomatic web software.

Anyone I follow that generates their own RSS feed usually does. A lot of CMS software doesn't, though. A Tumblr-hosted blog, for example, will cut off its RSS feed items if they include an explicit fold (i.e. a "Read More" link)—even though the intended purpose of those is to make posts not take up fifty pages on the Tumblr dashboard (the dashboard being a "limited preview" view), not to cut down or restrict the item itself.