How to Hack RSS to Reduce Information Overload | Hacker News Reader

How to Hack RSS to Reduce Information Overload

gigaom.com

16 pointskevruger10 comments

There seems to be two forms of 'news' that unfortunately rss readers try to do both of them. The first is feeds from sites that update 20+ times a day and the second is that blog that updates once every three months. The first is just there to read if you have the free time while the second you always want to read every single one. You can create two wildly different user interface optimized for the two modes rather than what we have today which is the best of neither.

I use different folders for the two types (Non-pro and Firehose (and professional for lead sites))

I hit mark as read on the firehose and lead site ones all the time.

Yep that's why I use two different readers for the first and second use...

In my opinion, RSS is "good" for one thing only, which is aggregating a bunch of rarely-updating blogs into a single location to check.

Following something like techcrunch.com in an RSS reader – let alone with unread counts – just seems like madness to me. I realize some people do it, though.

(Incidentally, this is precisely the problem my startup is trying to solve. You plug in your RSS feeds and it will assemble a personal "hacker news" of things you should be reading. iPad-only for now, though: http://zite.com)

I basically don't read anything online except through RSS feeds. My strategy for dealing with high-update sites is twofold. First off, I don't subscribe to redundant feeds (since many sites offer highly-overlapping content), and I am very trigger-happy on the unsubscribe button.

Secondly, I make use of this highly customized neural network-esque technology I use which gets a finely-grained history of everything I do and uses it to build a model of importance by which it can filter out irrelevant content. It's a bit processing intensive--it can only filter through 30-50 items per minute--but it's not a big deal since my RSS feeds only produce less than 500 items a day, so that's only 10 to 20 minutes of processing per day. It produces (on average) only a couple of false positives (items I don't actually care about) per day, and (although of course it's hard to empirically confirm) it seems to produce similarly few false negatives. This is a huge gain over every other system I've tried (especially populist voting-based systems).

I've evolved a Perl/Tk app (forked from http://cscrape.sourceforge.net/, but there's practically nothing left of the original cscrape code except GUI bits) that sniffs Craigslist RSS feeds and applies additional filters (beyond the limited search capability offered by CL) before presenting the results. Compared to reading CL RSS feeds in Google Reader, I'm definitely suffering less information overload. Maybe there are other RSS readers that offer such custom filtering, but this way I can add whatever feature I need (with full access to Perl facilities), when I need it. Also, though I cannot prove it, I suspect that basing this tool on standard RSS feeds is less burdensome to CL's servers than scraping the site ad by ad.

P.S. The XML in CL feeds makes the Perl XML parsers I've tried barf, so I had to write my own. Again I'm speculating but I suspect that (replacing the XML parser) would have been challenging to do in a "RSS Reader".

Edit: less pejorative comment on CL's intrinsic search capabilities.

I saw a similar post a few days ago but specifically for developers. Applying Service Oriented Design To Yourself: Information Stream Management: http://blog.darkhax.com/2011/03/22/applying-service-oriented...

I also built this tool to delay RSS feeds until after I finish work, you should check it out - http://rssafter5.appspot.com

For me, the biggest improvement in information overload has been simply unsubscribing from any blog that updates more than two or three times a day (Marco.org, keep; TechCrunch, remove).

Instead, I subscribe to their Twitter feed and any story important enough that I missed seems to find its way to me in other ways (e.g., here on HN).

I listen to my RSS feeds via FeedSpeak for Android : http://feedspeak.tk

Reply on news.ycombinator.com