Hacker News Reader

Comment by pbhjpbhj | Hacker News Reader

Do you just record n-grams indexed against the page url, are you then uploading that index? If you're not uploading it how is there no "local version" available?

It's an interesting idea. Personally I have a script that wgets all the pages I bookmark and I very rarely use that content. What use cases are you anticipating?

View on HN

It would be useful for pulling back pages that have fallen off the internet and are not in the wayback machine.

Thats a use case I have hit a few times. I even started backing up useful posts just in case they died.

http://gigablast.com/rants.html is an example. Its a really good insight into the creation of a search engine which really should be preserved.

One that did dissapear but has since come back is this post http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra... which I went looking for a few years ago but had dissapeared from all search indexes.

Pinboard offers an archival account (25$ a year) that works wonderfully for this. Take a look if you don't already know it.

https://pinboard.in/faq/#archiving

What does indexing mean here? I'm not sure what it means to 'index' but not 'store' webpages as the frontpage says.

I assumed they meant that they extracted meta-data about the page (page mentions "Winnie the Pooh" for example, like a book index) and kept that associated with a key (eg URL) but didn't actually store the content per se. I was hoping the answer to my question would include that sort of detail - what's stored and where.

That's exactly right. The indexed content is stored (as an inserted index) but not the plain text.

Reply on news.ycombinator.com