Hacker News Reader

Comment by me_again | Hacker News Reader

What I would really like is a little bit like this but not quite the same: full text search over everything I have ever seen on the computer. It would read and index the emails, web pages, word docs, etc as I open them, then later when I think "I know I saw a doc about cache oblivious algorithms", I can search for it without being distracted by 100K documents I haven't seen. Or I can find that email I read, without finding the same phrase in a bunch of junk mail I never opened.

Does anything remotely similar exist?

View on HN

The pieces exist, you can string them together with a Perl one-liner. You are interested in the set intersection of the following two topics:

Full indexing: <https://lesbonscomptes.com/recoll>, <https://userbase.kde.org/Akonadi>, <https://addons.mozilla.org/firefox/addon/falcon_extension> (If you're not content with a piece, then research substitutes on <https://alternativeto.net>.)

Recent: `.local/share/recently-used.xbel`

This does not help with the email part because email programs do not register opened messages in recently used. Work-around: install a DBus or AT-SPI hook and write your own database of recently opened messages.

Happy hacking!

Thank you for sharing this

I've seen this been called a 'personal search engine' before. One person who is well known for their personal search setup is thesephist[0]. My friend is also working on an extension that uses NLP to semantically index your browsing history so that any text on the internet can be turned into a hyperlink to something else you've read[1].

[0]: https://thesephist.com/posts/monocle/

[1]: (WIP) http://espial.uzpg.me

I've read comments from people (don't remember the forum, perhaps HN) where people have said that they did this. No idea if there's a public project for this that you can use, but people have definitely done it. I agree that it'd be useful to have, though you probably need a good way to filter out irrelevant stuff.

Try looking at karilicoss' promnesia and it's background for similar ideas and tools.

I’ve been working on exactly that! [0]

My info is in my hn profile, if you (or anyone reading) would like to chat about it.

[0] https://apse.io

For Windows, Google used to have something like that. Because it's Google, it has since been discontinued [1].

Mac's finder is close to what you have described, and works reasonably well for me.

On Unix, this sounds like something a grep one-liner (maybe with some document depacking/packing pipe for Office documents) would do.

[1] https://en.wikipedia.org/wiki/Google_Desktop

I considered doing this - take screenshots of your screen constantly, OCR them and index them. It's fairly simple. However, there are some problems

- OCR constantly running in the background is power consuming - What granularity do you take your screenshots? Imagine each screenshot is 500 Kb and you take one each second. This'd result in 40 gigs of data per day. How are we gonna store it? How many days data do you want to keep?

Privacy?

Since parent is taking power consumption and disk storage into consideration, it's fair to assume they are considering a local approach, meaning privacy is as good/bad as any other local data you have on disk today.

That's Apse – A Personal Search Engine https://news.ycombinator.com/item?id=27965979

There was once a thread here on hacker news about missing features of operating systems, as such a thing could only be achieved on OS level... can't find it anymore, unfortunately. It was mentioned that passwords etc. could be a nightmare. A feature like that would be a big dream of mine: Some kind of an individualized semantic archiving processing and a vector search engine to search through it.

Open-text question answering. Just make your own; index all paragraphs of all documents using TF-IDF as you access them, then when trying to search for something, use this index to get a set of candidate paragraphs and run them through BERT-QA trained on SQuAD v2. You can extend this to the content of images - first run image captioning using CNN and transformers, then index the resulting paragraphs the same way (in both cases, include a link to the original in the metadata). You might need to write some browser plugin/system driver to do it automatically as you access documents/images.

There used to be a project that kind if did this: https://beagle-project.org/. It's long since defunct and I'm always surprised that nothing emerged to fill the gap?

EDIT: I did a bit of Wikipedia rabbit holing only to discover that tracker [1] is currently running on my computer and indexing my files

[1]: https://en.wikipedia.org/wiki/Tracker_(search_software)

I think windows and Mac both do this by default, no? Just disable the web search and your local full text is what you're left with.

I believe they meant they want something that searches only content the user has personally directly accessed, not ALL local content.

Ordering by "Date Last Opened" or "Date Modified" does a fairly good job in some cases

This does introduce sorting contention though, what to sort for first? relevance or date accessed? Ideally you would want to introduce date accessed as an aspect of relevance itself.

Spotlight on Mac will show you recently opened, or frequently opened, files first.

I thought we all disabled access time (mounting with noatime) to avoid trashing SSDs so quickly.

It doesn't index text in local image files, and it doesn't index over the full text of all the webpages and epubs I've read.

For the web part, there's a tool called Recoll, and a browser plugin Recoll-we.

this has been built before. the problem is that it also needs attention for ranking

you know,,,, there was a april fools day annoucement on torrentfreak years ago, maybe a decade, it was describing this behaviour. that was nice

Reply on news.ycombinator.com