Hister: A private search engine for the pages you visit and the files you keep
github.com
github.com
Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites. It stores extracted content with offline result previews, so information remains searchable even when the original page changes or disappears. It supports full text and semantic search, can run entirely on your own machine, and includes a web interface, command line tools, and an MCP endpoint for assistant integrations.
Website: https://hister.org/
Tiny read-only demo: https://demo.hister.org/
Ps.: It looks like our name conflicts with a registered trademark in the US. The owner of the other project has asked us to change it, so we’ll probably need to comply sooner or later.
Name suggestions are welcome! Ideally, the new name should be relatively short, sound good, and have an available .org domain.
Thanks!
It sounds like history, storage and ethereal at the same time.
ιστορία, if you wanna copy paste.
and yes, the English word comes from the Greek word!
What does Hister do differently? Search seems like a major differentiator, I'm wondering if leveraging the existing archivebox project for archival and implementing good search on top would be more efficient
Edit: I RTFD - and it seems yes.
Looks like you can even set authentication up so you can run it at home but connect while you're away too...
Histeri
MyHister(i)
Hyster(y)
Also, I have been using your app for two months now. I have only had to rely on it a few times, but each time I did it worked beautifully. Thank you.
So? Where are you based? For what class was the trademark filed? When was it filed?
I doubt that he has any leverage, but I don't know the background.
How does it sync via several computers?
* chronilog.org ...as in, a log of one's chronicles.
* And if you will include this into KDE, then can use a 'k' instead, such as kronilog.org :-)
Both seem to be available. ;-)
When I have suggested names in other situations like this in the past, I spent the ~$10 to get the domain, and offered the transfer the free. Of course, not everyone would do this.
Question for you: For the less tech savvy of us on here, is there any chance Hister can be can hosted on something like Pikapods? https://www.pikapods.com/
E.g.: for this submission I would want both https://news.ycombinator.com/item?id=49743097 and https://github.com/asciimoo/hister captured.
Seekfold (seek and manifold)
Seekdex (seek and index)
It lets you search YouTube transcripts. If you could somehow integrate video transcripts into this tool, I would be extremely interested in trying it out
Put another way, i can't write an extractor for Reuters and then point a config to it from my current hister binary?
I ran it for most of this year but encountered some problems with it I couldn't fix and thus have not had it hooked up to anything since June when I finally couldn't take it anymore.
In short, I serve a good number of apps from an Nginx reverse proxy. Maybe 25% of them are exposed to the WWW while everything else is limited to the LAN but I still get valid TLS for all of it.
Hister, though, kept breaking my whole reverse proxy and I could never figure out EXACTLY why so I could fix it. After running fine for a few days, it would hog the whole server and everything else proxied by Nginx would become unreachable. I tried tuning the config for it to no avail.
One day when I'm less lazy, I'll probably hook it back up via it's LAN IP to every machine I've got again. I REALLY liked that I could log my browsing history from any machine anywhere in the world without a VPN and I was really disappointed when I had to disable its config in Nginx.
I still use it a lot to go find stuff I flagged as important quickly.
I'm curious if this is something you've heard of before, or if I've got a one off problem here.
I even ported the config to a brand new VM with NGINX and still had the same problem.
It might be something similar if all of your sites are subdomains. Try incognito at the same time next time
I've got some applications used daily by friends all over the world and the services would all become unavailable to them when this started happening. Only fix I found was restarting NGINX service and then it could happen again an hour later or 3 days later. Once I removed the proxy config for Hister from the service, the issue never happened again.
I'll repro the issue and get the details intoa. Github Issue this weekend.
And for the name, what about "Historex" (although already taken as well) o "Histearch", mesh-up of "history" and "search"?
"The HYSTER trademark is filed in the category of Education and Entertainment Services" [0], so you can safely ignore any demands to rename, as it doesn't conflict. Embarrassing for them that their lawyers don't understand even the basics of trademark law.
Edit to clarify as some folks here are as confused as sHyster's lawyers: Trademarks are not exclusive, they're restricted to a category or categories. You can be Apple in the category of computers, but not in music if there's already an Apple in that category (unless you have $500M to spare [2]).
[0] https://www.trademarkia.com/hyster-77843354
[1] https://tmsearch.uspto.gov/ for a more authoritative source than [0], but it doesn't allow deep linking
[2] https://en.wikipedia.org/wiki/Apple_Corps_v_Apple_ComputerOr more likely do, but hope that you don't...
I have it up on GitHub, but I don't think anyone should use my implementation.
Loosely, what I built:
* On each of my machines I have a cron job running that looks at all my web browser history (usualy it's inspecting the brower's SQLlite across firefox and chrome). If it matches my rule list: hacker news stories, certain reddits, etc. it'll grab the page, convert to markdown and drop in my Obsidian Vault incoming.
* It has a whole de-duping architecture since I might open the same page on multiple machines. Uses the CloudFlare SQLITE D1 storage for tracking the processed links.
* it'll then trigger the LLM to do some Karpathy wiki style taxonomy assignment to the articles, organize them, create an index etc.
It's then available for my "bot" stuff to do writings for me.... I will probably write more about it at some point. I'm not certain it's totally useful and not just a yak-shave on hoarding knowledge.
Ai-drafted article on this [1]
Example AI-Drafted article based on some discussions the other day on Ollma vs LLama.cpp [2]
[1] https://taude.xyz/posts/how-archivore-turns-browsing-into-a-...
[2] https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-o...
EDIT: it's also the type of thing that feels very personally customized for my needs. I encourage you to build something similar on the idea. Much like how Karpathy Wiki was suggestive and not a runtime to just use...
As far as the need for private search, well, I've already searched for or visited those pages, so...
The biggest win was the realization that both firefox and chrome maintain all the links you visit in a very queryable SQLite database. I've been poking at that for a lot of custom tools, like WHAT JIRA tickets am I paying attention to this week, etc....
I think browsers can play a part in building a local search index for URLs based on those keywords the page declares and cross verify/accept only those that are in prominently visible content, or may be delegate to an external engine(like LLMs) via an extension etc. This is particularly useful for cases where full text indexing is not feasible or desirable.
I doubt Google will ever add such feature in chrome though.
The poor data quality is a problem for anyone who wants to use the tags, which has seen everyone almost universally reaching for other solutions. Search engines have preferred anchor tags, bookmarking solutions have applied user tagging.
With keyword tags it's been a vicious circle of poor data quality and neglect since day one. Even in documents from the early 1990s when people were really trying, the data quality is inconsistent at best.
ditto, it's an experiment in near-vibe coding, which also uses Typesense for queries using BM-25 & RAG with fusion. I have the web search/fetch/crawl features persisting raw intermediate values (api responses, search result lists) because I might re-use them one day... at least good for auditability if I need to
related, it is using Hister author's prior project SearXNG as one of the search providers
Nobody seems to remember it, even though it was a headline feature. Was removed in 2013, I think due to technical constraints.
Will definitely try this.
Edit: Found it. Thanks Claude.
https://github.com/ssnangua/google-chrome-comic-hd/blob/main...
https://dfir.blog/history-index-files-removed-from-chrome-v3...
I think due to shareholders wanting new sportcars. The offline pages don't show Google Ads.
...And don't give me the tired "it was faster". this horse was already beaten to death and it was 110% marketing.
when it launched it was fast because it had LESS features.
The risks were clear from day one but Google and Chrome were great to both users and devs, and it stayed like that for a long time.
I even hacked it for a company I was working for at the time: I installed it in a machine that had a lot of pdfs from some other client company of them.
I don't recall the exact details, but I did some sort of proxying between the Microsoft Web Server that came with NT? at the time and G Desktop, and then an entire team of first support agents had almost instant search across all those docs. Good times.
It's like there's a timer or a cache somewhere, with aggressive limit, saying "only scan these top 100 results and then abort if it takes more than 10 ms to findf anything", or something. Making this completely useless.
That's on top of occasional deletion of browsing history past few weeks.
A while back I looked up a manual for an old Singer sewing machine, which turned out to be the model 15. So I typed [sin 15] in the address bar and -- yep, first result! A week ago I looked into the game Metroid Dread. Typed [met dre] and two relevant pages from my history came up, immediately.
I don't know what you mean by "100% unique match" but, this feature works for me, and in fact I regularly use it and find it very worthwhile.
I've been learning the eBay developer APIs. Recently (i.e. within the past few hours) I looked up the "Browse API" specifically. My first or second try failed so I typed just [ebay brows] and didn't find it! But when I finished [ebay browse] that document (within ebay.com and titled "Browse API | eBay Develo...") did show up!
I can't reproduce now so I can only guess but it's possible this was a timing thing. I was typing several things, and changing/re-typing "as soon as" I did or did not see what I wanted to show up. But there was definitely a moment where I'd expect "that page I just had open an hour ago" to be at the top, but instead search suggestions (ugh, search in the address bar ...) were all that was showing.
How do other people handle this dilemma?
Even solution I can think of involves are a great amount of extra work.
One thing that can be helpful when reasoning about things like this is figuring out what your actual threat model is. What does system compromise look like to you? Data exfiltration, arbitrary code execution, something else?
Unfortunately there's no one-size-fits-all solution for this yet, but there are a lot of groups attacking it from different angles: Qubes, GrapheneOS, Firejail, Bubblewrap, Android/iOS app permissions, Landlock, App Sandbox, etc.
We should be doing this with all software, regardless of provenance, anyway. Least privilege applies to servers just as much as it does to users. Even if the software isn't untrustworthy you can be it has vulnerabilities.
My first go-to is network segmentation because I spend most of my time doing networking work. For every vendor who has shit-talked me to Customers ("Wah, wah! Your networking vendor is making this so much harder because they want us to enumerate our traffic!") I have concrete examples I can cite when attacks were stopped by network segmentation (preventing shellcode from downloading a payload, preventing C2 communication, firing off alerts when unexpected network traffic starts coming out of a host, etc).
Beyond network segmentation, I am very suspicious of software that needs to run as a privileged user. So many attacks get easier when privilege escalation in the host OS is already done for you.
Apart from high-overhead solutions like VMs and containers, there are seamless and maintenance-free solutions (after the initial setup):
- systemd service hardening [0] [1]
pretty powerful, but it's a blacklist approach - whack-a-mole
- AppArmor [2]
Whitelist, proactive approach. Contrary to SElinux, it's not a programming language, and could be grasped pretty quickly. I made a tool to easily convert AA logs into usable rules. [3]
[0] https://github.com/alegrey91/systemd-service-hardening
[1] https://github.com/desbma/shh
For me: consider this a form of paranoia and ignore it, while worrying more about cleanup costs of non-vetted packages.
Like, even if there is 5% chance that a program I download will start downloading global python or node packages, odds are within a year I'll deal with couple that have mutually incompatible requirements and are impossible to run without more VM surgery than I have patience for, and that I'll discover this only after a botched installation bricks software that used to work before.
But that's solvable with less extra work. Just throwaway containers. With no hand-wringing about read-only access or isolating it from network, because my threat modeling doesn't consider loss of privacy or any data leak from my personal local side to be realistic or impactful threat event (I mean, everything that matters is in someone else's cloud and by design not trivial to access) - OTOH, it assigns great magnitudes to loss of personal time.
I built myself a little extension last year that tracks what information I was looking at, but focused on generating "new info" recaps for the day / week.
I realized that I open / quick view a lot of pages and close them, which is a strong signal that I don't care about that specific page, and it shouldn't be a source of "new insights" that I learnt that day (since I probably don't care about that topic).
I'd love to re-try a simpler version of that project that builds on Hister as a backend actually.
Adding a timer constraint would be really nice and I guess it might be quite easy to do it since it's only a concern in the browser extension.
This has an MCP server specifically so a workflow like that would work for you. This is just made to gold the data, and I'm a human accessible way should your AI fail you
You didn't solve the problem, you are just trading pain points.
https://github.com/gildas-lormeau/SingleFile
"SingleFile helps you to save a complete web page into a single HTML file. SingleFile is a Web Extension (and a CLI tool) compatible with Chrome, Firefox (Desktop and Mobile), Microsoft Edge, Safari, Vivaldi, Brave, Waterfox, Yandex browser, and Opera."
1. Leaves all kinds of css, image file, which might be hundreds for each saved page. SingleFile saves everything in a convenient, working, single file. 2. is not really saving all the info.
I got burned when I was using Mozilla/Internet Explorer to save mhtml files, which ended up not being supported anymore. Additionally, turns out, they were saving the original versions of pages, without the Javascript changes, meaning that all the opened threads of comments I wanted to save were never saved! Never again!
If you check what kind of stuff SingleFile does, you will find a list of things Firefox is not doing.
Can I add NOTES about pages? This might be a good spot to do that...? Maybe the interface can be in a web page instead of terminal?
Before Google took off there was a vibrant ecosystem of FOSS dev around search, all different little aspects of it. Then after Google people stopped fiddling with search, search became "solved" or maybe "must be coded by the big boys". Shame.
Thank you for this, looooong time coming
Notes are not supported yet, only labels. But it is a useful addition, added to my TODO.
(For file search, I use voidtools' Everything, and I'm not sure why anyone other than Microsoft would want to mix local file results and web results)
This project looks cool, though I misread the name as "Hipster" initially [sips almond milk macha coffee]
For person using resource-constrained computers where CPU, memory and storage space is limited
URLs from the local forward proxy log are extracted periodically and stored in compressed files (URL logs)
(I also store post-data)
The compression method used is old and unpopular: recursive pairing
Compression ratio is better than gzip but worse than zstd, compression/decompression speed better than zstd but worse than gzip
More recently a method was developed to search these compressed files
Size of compression utility: 42.3K static binary
Size of search utility: 102.4K static binary
No Java
Limitations include basic regex only (no back-references) and files must be line-oriented
No decompression step is needed. IME, this search is very fast. If it is slow then this means the keyword is too common: refine the search
With minor modification (insert a newline at the top of file) I can also search inside compressed tar files
As a www user with underpowered computers doing relatively small jobs, these old, unpopular methods have proven to be fast and reliable for me
When I'm searching more than just URL strings, e.g., dates, titles, etc., I reformat the data into SQL and store it in a text file
Instead of storing large SQL database files, I store the text file compressed with recusrive pairing
I can then search the compressed SQL using basic regex; the output is piped into sqlite3 to create a "results" SQL database, e.g., in memory
For me, the speed of sqlite3 in creating relatively small databases is excellent
Then I can query the "results.db" using SQL
I like the search ui. my projects become functional but never polished. https://github.com/sbeckeriv/memoir
I also have it index my Obsidian notes, which is another little bonus for global search.
I did need to build up quite a few exclusion rules early on, but it's been hands off since.
[1] https://git.brucewillis.sexy/~tombert/fs_index I promise, safe for work, despite the URL.
I like it.
Is there any site/project that works as a fully customizable personal front-end to all other SERPs?
When I search for something, I always want a link to the best Wikipedia result. This should always be in the same place and have a giant icon/picture.
Then there could be easily clickable links to the SERP pages for Google, DDG, etc. for that query.
A big link to route it to your favorite LLM.
Seems like you could have a really useful "homepage" for all searches that sat in front of all the other sites. It could be local only and would not require indexing the web. Also wouldn't be a files search thing, as Hister appears to be.
Every single web page I’ve found interesting, since the advent of the Web, I have printed to PDF and stored locally for my own personal reference.
Something like 80,000+ files - my own copy of my own Internet - indexable, searchable.
Available offline. Something to read when I am far out to sea.
There is no need to involve third parties in your Internet history - no matter how trustworthy they seem to want to appear.
Print to PDF, and you’ve got everything you need, safe and sound.
The more "modern" the site, the worse it is. Surprisingly, government websites often print correctly since they've done the least amount of work to make the site modern looking.
Even full page screenshot doesn’t always capture the non visible part of the page (below the viewport).
Reader mode.
$ pdfgrep -r -i -n -H "your mom" ~/PDFArchives/
Does Hister handle this well? If not, can anyone suggest other options?
One of the most useful use cases for me, especially since I work in GTM, is keeping track of new ways to get replies from prospects, whether through cold outbound or things like SEO/GEO optimization. I read at least 2-3 articles a day on this, and it genuinely helps me figure out which ideas are worth trying because I can now keep track of everything I’ve read.
Running this would be nice but it still takes time and still there is no ROI for me.
Of course there will be people who find it useful but I am pretty much done with building knowledge bases or having todo lists.
Stuff that I need to do or remember - everything else if I forget nothing happens and it doesn’t impact my life or work.
https://www.history.co.uk/articles/did-nostradamus-really-pr...
It's badly needed, and so far it's working well for me.
Of course, it's impossible to know for sure what was LLM processed or not, but some of your posts (like this one) have been getting classified that way.