Hister – A private, full content search index that you control

would've been great with a more liberal license

Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks and archives and I've kinda been stuck grepping them at best (it's rare but I do sometimes want a page I saw once three years ago and I love having that option), while hoping someone would build something better.

One question if ya don't mind, while I explore: any chance of singlefile support? Content-extraction is useful in lots of situations (e.g. wallabag) and it's a great default, but sometimes it fails and sometimes you really do want the page, relatively close to how it actually was. Singlefile does that much better than most, and it does so well enough (and manually-handle-able enough if needed) that I don't feel any desire to switch to WARCs or similar.

Though specifically I'm probably looking for something like "content-extract everything" + "key combo to save singlefile version too" + "upload singlefile archives to backfill / recover". Like 99% of the time content extraction is preferred, and I'm glad to see it... it's just not always enough, and having to go elsewhere for exceptions breaks a lot of the utility.

I often find myself irritated because I read an article on my phone 6 months ago and the history is gone.

Would Hister be suitable for this? Can it index mbox files? Would it handle this amount of data? Does it have a search API so I can build an MCP server?

One thing at first look, like a little feature request ;).

Personally I think it would be very nice if it was possible to index different sites to different search indexs. So you could separate different stuff, like job, different projects you have, and other stuff and then last everything else etc to there on search index db.

So it would be possible to have like "profiles" you could easy choose from. So in the addon, when you click on the hister icon on the toolbar there would be a list of profiles that you could choose where to index the site to. But also a setting to choose which domains and sites that always should index to one search index profils. And all sites that dont is added to a profile is added to the default index instead.

For me this would make it much more easy to find stuff and sort out what Im looking for (for me things get very "thing"/project based what I want to find), so having this..

1 - The "profiles" would become like an important filter. When you search you could easy mark one or more "profiles" which you would search from, and you remove alot of unwanted data automatically, specially when you index every site you visit by automatically.

2 - It would also make it easy if the db gets big over time to remove data that is less important later on and that you dont need to be index anymore.

3 - It would also make it easier to backup only the most important search index DB:s if the DB for the differnt search index:s where in different folders/files. I guess the "default index" that index all sites could easy get big, but there it would be alot of not important data. So would be nice to be able to easy just use a backup program and only backup the only important search databases, to save space.

Thanks

Academics usually take it to mean the Danube river instead, but in conspiracy/New Age contexts, the association with Hitler is prevalent, and mentioned in several pop culture works, including at least one feature film.

I'm hacking on a in-process DB on top of LMDB+Lance(for now, hilbert space kung-fu with a custom matryoshka embedding setup with separate spatial + temporal + internal and content derived anchors will replace that ~last-century~ last-year tech) + roaring bitmaps as the primary indexing engine with the same or similar goal[0] and will definitely deep-dive into yours.

I'm also trying to index users unstructured documents and workflows(tabs, emails, files, notes, identities etc)

- Organize them into semantically meaningful user or agent created context or directory-like virtual trees (the same photo of a nice kitchen may be surfaced under `/travel/barcelona` and `/arch/interieour/kitches`)

- ..where tree nodes are mapped to bitmaps - `/travel/barcelona` does a fast and cheap `travel` AND `barcelona`, want to "zoom-out" you just go one directory up to `/travel` and see all documents tagged with travel)

- You can use multiple timelines - extract that fancy md-converted en-wiki hf dataset into a wikipedia db dataset + timeline, tag your personal timeline as "personal" - wanna know the zeitgeist of your grandmothers birth date - search for it with timelines personal + wikipedia in layered mode and you'll get everything that happened or was happening during that time.

- You can have long-running stateful query sessions and refine your searches dynamically - search for "winter" and get all documents with a winter scenery or mentioning winter - refine with "nice view" then "laptop" - citing a recent example[1]

- Documents have relations that would be cumbersome to map in a virtual tree structure(worth an experiment due to the zoom-in/out you get with context bitmap trees though) - hence on top of the initial structure you can use graph edges(also powered by bitmaps - as most indexes are)

- All vector queries always run on top of a candidate set you get by the bitmap/bitmap-based filter algebra hence searching through 100k+ docs is usually pretty fast

Anyhow, let me stop here, thank you once again!

[0] https://github.com/canvas-ui/canvas-synapsd (sorry for the sloppy ai readme, no time to resurrect my old one with the updated APIs)

[1] https://demo.cnvs.ai/pub/c/aks6zaf8

What tools or features would Hister need to support your complete search workflow?

It also has a "public mode" where anyone can search the indexed content, but only authenticated users can add or modify it.

Also indexed data is persisted on a per-user basis, so you got this isolation and certainty that your searches will not be polluted by your family's

Others could be answered by examining the source code, as it is open source.

It's good etiquette to check existing documentation and resources before badgering OSS project teams with a lengthy list of questions.

I feel this is a weird spin on the "Clicks to Hitler" game

It's a terrible choice of name