Hacker News Reader

Comment by AngryParsley | Hacker News Reader

I used to work on Bloglines. It's sad to see it finally die, but the writing was on the wall for years.

There are two questions I'd like Ask.com to answer: What will happen to the source code? and What will happen to the archive of RSS items?

Ideally, Ask.com would open source Bloglines and get the item data over to archive.org. We never deleted anything, so the item archive is around 11TB and has blog posts going back to 2003. I bet quite a bit of that data doesn't exist anywhere else. Although Sturgeon's Law applies, there are some gems in that data. I took a look once and found some interesting blog posts from the 2004 election cycle and Hurricane Katrina.

I seriously doubt that will happen though. Building an archival tool for the podserver/itemdbs would require more dev work, and there are the hardware costs of reliably storing 11TB.

View on HN

I hope not - I had private RSS feed from things I don't want to share with the world, and I can't be the only one.

Bloglines already has that logic. It would be built into any archival tool as well. If the feed url scheme is https or if it's using http auth, it's private.

What might it cost to buy off the data to a reliable datastore? Could the format of the data be opened to let outsiders write the processing tools?

No idea and probably not. The Bloglines podserver/itemdb was built internally. You can get some idea of the setup by reading the slides from pquerna's GeekSession's talk: http://journal.paul.querna.org/wp-content/uploads/2007/10/bl...

11TB can be reliably stored on a 2U server with 8x2TB drives. Figure $150 per drive = $1200 plus cost of the server.

> I bet quite a bit of that data doesn't exist anywhere else.

You'd really put money on that?

I'd put everything I own on it (for reasonable interpretations of 'quite a bit'). I know from personal experience that web pages and blogs die like flies, and that the Internet Archive - the only live possibility for pages in general - often simply doesn't have copies or the copies are unavailable. (Lost a number of _Mainichi Shimbun_ articles, IIRC, to a robots.txt that went up years after the articles did; the IA respects it anyway. Sometimes the domain changes hands and the new owners put up a restrictive robots.txt; I hate that too.)

11TB of HTML & images goes a very long way. To put that in perspective, that's in the same area as a full copy of all Wikimedia projects, including the English Wikipedia and Commons.

It seems to have firsthand knowledge that this data has been stored permanently by Bloglines; what I'm curious about is why you think it wasn't stored by any other companies that crawl the web for a living.

If a tree falls in Google's archives and no non-Googler can access it, does it really exist? Anyway.

As AngryParsley says, the specific Google services in question inherently have a gap of a few years. I have already given examples from my personal trifling experiences looking up a few hundred dead sites where the sole public archive, the IA, fails. Scale that up to hundreds of thousands and millions of sites...

It's just HTML/text and it's 11TB gzipped as of 2008. I bet the archive has grown since then, but the valuable data is the oldest. Uncompressed it's probably in the 20TB range.

I'd also bet everything I own that a significant fraction of the data is unavailable anywhere else. Google Reader started in 2005. Ditto for Google Blog Search. Even if they crawled the same feeds as Bloglines and kept the data indefinitely, they'll still be missing those first two years.

> Uncompressed it's probably in the 20TB range.

That seems a little low to me if you're starting with an 11TB archive. Doesn't gzip usually shrink HTML/text down to more like 1/5 or 1/10 the size of the original, rather than less than 1/2?

That greatly depends on the nature of the original data. Some forms of data are already compressed, such as images. Most websites apply gzip or DEFLATE (gzip is built on DEFLATE) compression serverside if your browser indicates it understands it.

Reply on news.ycombinator.com