Hacker News Reader

Comment by gambler | Hacker News Reader

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all.

The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of this sort. Unfortunately, the most popular browser in the world is controlled by the same people who control our protocols.

If I ever get the energy, I will attempt to write a browser extension for this.

View on HN

Shameless plug: I built an app for that. It’s not super polished but it is at a point where I use it to store technical documentation that doesn’t ship with the software (aka docs for every Javascript package I use). It works for moderately complex JavaScript apps as well.

https://github.com/CGamesPlay/chronicler

Check out https://blog.webmemex.org/. It serializes pages, inlining their external resources, full-text indexes them, and saves them locally. Browser extensions are available for Firefox and Chrome.

You're describing IPFS pinning.

https://docs.ipfs.io/guides/concepts/pinning/

Best use of time and money is probably http://pinboard.in/ ?

Seems like someone I can trust not to sell out and shut down.

But if they die?

...or just run out of money?

Storage must be local, physicslly user-controlled. All online stuff is a backup only.

On top of this there is currently a push for a mix of censorship and 'compulsory removal' on the internet. For instance the Canadian Supreme Court ruling Canada can force Google to remove worldwide results for something, the UK censoring increasingly arbitrary things, or various 'right to be forgotten' type laws. And of course there are also intellectual property lobbyists and special interests pushing for ever more draconian rules on that front.

Any centralized online site that aggregates other data has an, at best, unclear future ahead of it. Really though, centralized is the key word. The internet so desperately needs to migrate to decentralization but we're collectively about as inertial as an 882 foot long, 46,328 ton tanker, built in 1909.

why cant backup be local, user controlled?

revenue looks like hes doing just fine. https://blog.pinboard.in/2017/07/eight_years_of_victory/

https://news.ycombinator.com/user?id=idlewords

Saving data locally is not a protocol problem.

Chromium is almost there.

Chrome had “chrome://flags/#show-saved-copy” which would do you want if you could make Chrome's cache arbitrarily large and persistent, but it appears to be gone now.

Check out Beaker Browser. Solves this problem on the protocol level and bookmarking automatically stores a copy.

Use the wayback machine browser plugin allowing you to conveniently use the "save now" function which immediately saves the website to archive.org https://archive.org/web/

There used to be a Firefox extension called Scrapbook. I think it was one of the winners of one of the two first extension competitions.

It could easily download entire sites or just a subtree, including resources. It would the rewrite URLs as necessary to point to the downloaded documents instead of pointing to absolute URLs on tje Internet.

Still hoping that Mozilla will improve its extension API. I mean I can see the reason for not allowing everyone to pone around everywhere under the hood, but the current API is rather anemic it seems.

Been using it for so long. Currently it's Scrapbook X under Pale Moon (that I keep around just to run that).

Looking for an alternative that would import my thousands of saved pages.

Maybe I don't understand what you are saying, but I safe webpages all the time. Safari the creates a .webarchive file with everything in it. Isn't that what you want?

No. Because that's extra work. It should just be automatically done whenever I bookmark a page. In fact with disks so big now I could just dedicate a terabyte to just caching everything I visit. Make that indexed and searchable and a lot of what I use would be forever available offline. Pages could expire automatically if unvisited for some period of time, perhaps weighted by the number of times they were visited.

Perhaps I should try to resurrect my rusty and incomplete Firefox add-on skills.

https://historio.us

AFAIK there’s no self hosted equivalent. Ive been using it for years. Whenever I put a link in my docs / notes I use that site to cache a copy as well.

When I first signed up I wrote a script to convert my bookmarks to a text list so I could import them. Around 25% of the pages I’d bookmarked were gone.

Tangentially related, but the reader view of a page seems ideally suited for archival bookmarking. It's unfortunate that so many sites seem to be intentionally breaking the reader view functionality in order to maximize newsletter subscriptions and eyeball monetization. Recipe sites are the worst offenders IMO.

Pocket seems to have luck where other reader views and outlining services fail, and you can set up IFTTT to download pocket articles as html files to dropbox, but that's certainly not automatic and completely out of your way. I would have preferred it to archive as a plain text document but I couldn't figure it out through IFTTT.

Are the downloaded pages in original or Pocket/Reader view?

You can export (with some pain) your Pocket list. Fetching sources from there is ... somewhat painful. I've done it a couple of times, but it's not something I relish doing.

Pocket Premium stores the web pages on a server. If there was a way to batch export and access them locally, I'd gladly pay for it.

I’ve been telling people this for years. Bookmarking is for dynamic data. You bookmark the weather website. You don’t bookmark recipes. Personally I use Evernote and its web clipper. Unfortunately it doesn’t download images, but I wrote a little program using their API to replace the img tags with embedded resources.

wget -r ?

No. There are some cases where it is useful to download many pages in a batch, but what I am talking about is, effectively, partial local replication. Bookmarking I describe should create a tiny (but useful) version of the web on your computer.

It needs to be seamless. It needs to be searchable. It would be incredibly useful if it would capture relationships between pages (links) in addition to pages themselves to navigate offline.

I can see many use cases for a kind of "super bookmark mode" where the browser automatically stores all the pages you visit within a certain domain.

I would loooove this. Open one of your bookmarks and you can't tell it's the local copy unless you looked at the local path sitting in the URL bar. Maybe make an option to fetch a live copy though for sites like the front page of HN that are in constant turnover, but it would be great. Storage has never been cheaper, let me hoard!

You can even take it a step further and create a viewable timeline of all the pages you've visited in case you have something at the tip of your tongue, but need to retrace your steps and logic to get there. Browsing history is kinda lackluster for this. My last ten entries on firefox are three hackernews articles with "Add Comment" dispersed half a dozen times in that list. If I had something along the lines of zoteros timeline for papers I could probably find stuff way easier.

HTTrack does most of that, minus "seamless". You could probably write an extension that automatically sent bookmarked urls to HTTrack for mirroring/archival, then they would all be browsable offline.

You can't seamlessly replicate a REST server.

Don't forget --warc! :-)

https://en.wikipedia.org/wiki/Web_ARChive

Reply on news.ycombinator.com