A sysadmin's rant about feed readers and crawlers

(I want to preface this comment by saying I don't mean in any way to rebut the thrust of the article, I worried this could be taken that way, rather it's a commentary on the inconsistency of feed implementations that I don't find discussed much.)

As a somewhat orthogonal observation, I've found it's surprisingly hard to write a crawler that's well behaved with consistent heuristics across a variety of different feed providers.

Usage of published vs pubdate vs updated, which is changed when, (or all three just containing 1970 and/or the current time), reordered feeds, items published out of order _regardless_ of the scheme used, changing urls/ids, etc. Whatever set of heuristics one uses for some sites may not apply to others. Now, this begs the question of "why not make parameterize and tune", and yes, this is largely what I've resolved to, but it's, to the core point, more of a moving target than one would expect from how ostensibly simple RSS is.

At the end of the day, in many cases I just fall back to using a cache of recently-seen-urls and, when possible, short-circuit the enumeration when I cross one. (Similar disclaimer, I do love me some RSS, I've just never had a good opportunity to rant about this.)

The problem is there are ill-behaved feeds, which need ill-behaved crawlers to work right. I mean, not all of the ill-behavior she mentions is caused by ill-behaved feeds, but that does include scraping individual posts. That's actually my biggest problem with feeds, is them deciding not just to not include the full post, but to not include any of the post. So all I really get is that they have a new thing.

Pulling articles referenced by feed does make sense on corporate press releases pages that are handled by PR Newswire - they do not put actual content in the feed, only a link.

I'm just going to echo the other comments here: All of this is good sense when the publishing site is well-behaved, but that's rare enough that I'm completely sympathetic to readers doing weird stuff to get around extremely common ways people break their feeds. Actually putting the full content of your pages in the RSS feed is, in my experience, outright rare; I generally want my reader to ignore the barely-one-paragraph snippet in the feed and just go fetch the real page from the site. Caching/Last-Modified is great until the site lies and your reader proceeds to never show updated articles. Pretending to be GoogleBot was, for a while, a great way to bypass paywalls and other shenanigans.

Granted, updating every 10s is really stupid. The only way I could imagine that being sane(ish) is using RSS as a generic protocol for plumbing automated data movement rather than for fetching actual blog posts. Daily is easily fine for 99% of feeds that are not themselves aggregators.