Comment by mananaysiempre | Hacker News Reader

Comment by mananaysiempre | Hacker News Reader

Parent Full thread

> my computer should search across everything I’ve read and some small high-quality subset of the Web

Even without the contextual search—which is initeresting mind you!—I want this part, and I’ve wanted it for a long long time.

I have it for journal articles simply by the virtue of obsessively saving and filing every PDF I’ve read for the last decade, but for the Web I don’t know how I could just plain index everything I’ve read (and not just everything in Pocket/Wallabag/etc). For example, is there anything that can pull links from Firefox Sync and save/index extracted text from them in a halfway reliable fashion? (I don’t expect it to be able to bypass yucky stuff like CAPTCHAs on archive.is.)

I know there are a handful of WARC-ecosystem tools, but AFAIU these are rarely that automatic, are tied to a browser plugin as opposed to just pulling the URLs from history sync (yes, I’m unreasonably fond of Epiphany/GNOME Web), and kind of heavy. Am I wrong? Any alternatives?

https://chromewebstore.google.com/detail/habonpimjphpdnmcfka... (or https://github.com/tjhorner/archivebox-exporter for source)

Pushes your history to ArchiveBox, which does the heavy lifting storing/processing the content.

Alas, might not work with Epiphany because there's no complete extension support.

But IIRC, it stores its urls in $XDG_DATA_HOME/epiphany/ephy-history.db - so a bit of sqlite and ArchiveBox might do the trick for you.

Note: I'm running something similar, but find that I'd rather not rely on my history, I tend to click on a lot of garbage ;) You might want to curate a bit.

> plain index everything I’ve read

Recoll together with the extension can be configured to index every single web page you browse.

https://addons.mozilla.org/en/firefox/addon/recoll-we/

I built something like this over a weekend. Saves every page title, html body, and time and lets you search them in the chrome new tab page. It also indexes on the fly.

I have it installed on my browser and I’ll probably release it when I can figure out how to make it more useful.

There's a different strategy, it's a VC thing that's mac only but pretty fun to play with:

http://rewind.ai takes screenshots continuously and uses mac's system OCR to index everything you've ever looked at, whether that's web, code, or messages. When you search it pulls up screenshots that match.

I've never heard of this service but am intrigued. Also, props to the Rewind team for putting privacy front and center. (Now I just have to decide if I trust what they're saying.)

Reply on news.ycombinator.com