Hacker News Reader

Comment by graderjs | Hacker News Reader

Hey. My project Diskernet does this: full text search over browser history.

Put it in "save" mode when using Chrome (linux is fine) and it automatically saves every page you browse (so you can read it offline), and also indexes it for full text search. It's a work in progress and there are bugs (so my advice initialize a git repo in your archive directory, and make regular syncs to a remote in case of failure -- that also gives you a nice snapshotted archive).

Anyway, best of luck to you! :)

Diskernet: https://github.com/crisdosyago/Diskernet

View on HN

>22120 is licensed under Polyform Strict License 1.0.0 (no modification, no distribution).

So this is basically Freeware / Shareware in the old days but with Source available? i.e not Open Source as defined by OSS. Do we have a term for it? Shared Source? I know Microsoft tried to use it but it was early 2000 anything M$ did at the time were extremely unpopular.

Will be trying this out soon because I just realise either half of the internet in the past 10 years have eroded, or Google simply cant search something I am sure I read about it 6-7 years ago.

I've always heard this refered to as source available.

The claims are impressive: that the network requests are saved, not the document after the page is loaded. I will certainly give it a try!

One downside: only works with Chrome-based browsers. Perhaps a way to implement this more generally is as a proxy server.

How does it work with a JS heavy app?

With an SPA the article might not appear on initial request, but later requests would be JSON which presumably you don't want to return in a search

It doesn't index each resource. Only the page text of the URLs you actually navigate to in a tab.

So this is fine. It won't index a JSON response in an SPA, as search result (tho it does save it), but it will index (and re index) the actual page content, as you browse it and as it updates (eg dynamically loads in an SPA).

Try it out, you'll have an easier time understanding how it works (as long as the current tag release works, heh :) if not try a previous tag).

A screencast would be tremendously helpful!

Good point.

The problem with a proxy server implementation is modern browsers (AFAIK) are unwilling to submit HTTPS requests in the clear to a proxy server rather than use CONNECT. There’s nothing about the protocol that would make that impossible or even inconvenient, browsers are just unwilling to do it. And I can see the reasoning, but if you actually want this to happen, like here, you’re stuck (or have to MITM yourself, which is its own can of worms).

That's the whole purpose behind ZAP and I use it for archiving pages all the time (they use hsqldb as the file format); it works fantastic for that purpose, but does -- as you correctly pointed out -- require MITM-ing the browser to trust their locally generated CA: https://github.com/zaproxy/zaproxy#readme

Thanks for the reference! I never investigated ZAP closely, for some reason it never occured to me it might be able to be used like that (if anything I’d have turned to mitmproxy, but that would require building a substantial amount of stuff to handle the actual archiving).

The problem with MITMing your own browser is (apart from the fact that it is an ugly hack in a security-critical portion of your setup) I don’t think any tool for doing that (including the one you referenced, from what I can find quickly) applies the complex set of important stuff browsers do on top of just verifying chains against a root store.

The bare minimum for me would be HSTS and the HSTS preload list, but I’d also like to see CT and Must-Staple enforcement, OneCRL support, TLD and validity term restrictions for some roots, and so on. (This is more or less what Chrome does from what I know, though I think they have their own equivalent to Mozilla’s OneCRL.)

Given what ZAP is designed to do, I'd bet $1 it will actively strip off any such headers before returning the response to the browser, since I think by definition injecting a MITM cert is the very case such stapling is designed to prevent against :-)

But, corporate proxies must face similar problems since they, too, MITM things, but I'm deeply thankful that I don't work in such an environment in order to know what the behavior is in that circumstance

I hope this doesn't come across as glib, but ZAP is Apache licensed, so if you are able to come up with the security behavior you want, I'd bet they'd welcome any patches to help implement it

Looks awesome! I see you came up with a new file-format to store responses; have you see the webbundle spec: https://wicg.github.io/webpackage/draft-yasskin-wpack-bundle...?

On the archival (rather than authoring) side, the usual format is WARC[1], preserving the complete content and timing of all HTTP requests and responses, but the tooling is clunky to put it mildly.

[1] https://iipc.github.io/warc-specifications/

This is a great project! I have, over the years accumulated a bunch of bookmarks (exported format) and haven't found an easy way to save "snapshots" for them. This fits my use case perfectly without having to roll out my own tool.

I've got a very different goal than you, but with quite a bit of overlap on saving web pages for historical lookup.

How are you pulling and storing the pages?

For context; I feel like there are quite a lot of solutions here, but not many overlapping and often reinventing many wheels. Memes aside (14->15 standards), i wonder if there's a way i can write my version of this that would benefit others by using common formats we can all benefit from.

edit: Interesting, sounds like it's network/proxy request based. Perhaps there's minimal overlap here?

Hey. Any chance this can be used with another chrome based browser such as Brave?

Should be fine. But you may need to fiddle with how the browser binary location is discovered, because we start it automatically.

Anyone who has used this for a significant amount of time can tell me how much gigabytes you've used to store your browser history along with how long you've been using it?

Does it download any blobs (from you) while installing via npm?

I don't think so. It just installs dependencies via npm. If you noticed something amiss please email me

Reply on news.ycombinator.com