Show HN: Save and search the full text of your browser history
browserparrot.com
browserparrot.com
We're rethinking how we interact with and manage all the data in our lives. Data is all over the place now (computer, mobile, multiple cloud providers...) and it's more than a bit unwieldy. Not only is it unwieldy, you can also lose access to your data at the drop of a hat due to an outage, a policy change, or a bug. This doesn't sit well with us.
Of course, to really tackle this problem will be the work of years, so we're starting with something more manageable. We've created a tool to aggregate all your browser history and make it full-text searchable. If you've read something online, you should be able to find it again—quickly.
There's plenty of prior art in the search-your-own-history space, so why might you care? When we tried to find an existing product to fit our needs we were unable to find something that checks all of these boxes:
- All history, from all browsers.
- Fast. It should feel more like autocomplete than like web search.
- Completely local. No need to trust someone's server.
- No user input required. I.e. no need to bookmark. Store everything.
- No limits. Don't delete my history unless I say so (browsers will delete your history after a few months, 4 months for Chrome).
I'm sure we've missed a few though. How do you keep track of everything you've read online? Any workflows or killer hacks you've discovered?
edit: formatting
Does the electron GUI need to be open for it to fetch the data, or can it do that as a background service as well, for example if one just wanted an always up to date persistory.db to use directly? For example an api server and a custom browser extension consuming it, or are these also being considered for browserparrot
Regarding the GUI, good question. The GUI does _not_ need to be open. There's a background process that periodically copies new history from the various sqlite databases into the BrowserParrot db.
In Activity Monitor this process appears as "core." I know, terribly ambiguous name. It was originally named for the Clojure namespace where that bit of code lives, but renaming the process to make it more clear what's running is on our backlog.
Can also find the process via `lsof -i :44444`.
But please release a linux version ASAP :)
I’m also not sure if I like that is’s always running in the background, not because of privacy but because it just seems a bit unnecessary - something that I use maybe a few times in a week and that’s nothing more than a search engine shouldn’t be using system resources all the time
I often use Cmd-tab to switch between apps on the mac, and I noticed that this search pane shows up in the tab switcher, which can get in the way a bit.
* In vscode, cmd-tab to chrome
* cmd-E to search for something
* cmd-E to close the search (goes back to chrome)
* cmd-tab to go back to vscode (but this goes to browserparrot)
Alfred, a similar search app doesn't do this, and I quite like that behaviour. I found out how to change this, as a user, here, but it would be nice if it was done by default: https://apple.stackexchange.com/questions/92004/is-there-a-w...
Strongly agree. Something like this is on our todo list. Our thinking is an optionally-open, in-the-dock UI to manage settings and a floating non-dock search bar for quick access. Perhaps with a menu bar item to make it clear something is still running.
So the dock app could be closed when not in use, which is would probably be most of the time.
We’ll be adding an option to run in the taskbar rather than as a full-fledged app-with-dock-icon soon to help with this —- being able to do everything fast and from the keyboard (including switching apps) is high priority for us
Do you guys plan on providing similar client-side search tools?
I’d love to be able to search my emails, notes, or third party services locally. Neeva is a search engine that provides this capability, but you need to grant them access to all your third party services so they can do the processing on their servers. YUCK!
So yes, we want to search everything locally. For me personally I find Notion, Roam and _all_ chat apps to be particular pain points.
Email too, although I've found that email search often delivers what I'm looking for so the pain point is much lower.
We haven’t added a UI for it yet, but you can actually change the hotkey by editing ~/.config/persistory/config.edn and changing the :quick_search entry (CommandOrControl+E by default) to your desired keyboard binding.
This functionality comes from electron, more info on the hotkey syntax in their docs [0]. Definitely don’t want to be getting in the way of your PyCharming!
[0]: https://www.electronjs.org/docs/api/accelerator#available-mo...
Not sure, but I think they only search pages that you explicitly bookmark, unlike BrowserParrot
The exception is Safari, which lives at ~/Library/Safari and requires Full Disk Access, granted via System Preferences, in order to be read.
- Show the date range from the earliest bookmark to the latest
- Let me open the downloaded content for a URL instead of redirecting me to the current online version.
I'm also wondering if the downloaded "extract" is somewhat complete. For example, reader mode doesn't really work on HN or Reddit, but the individual comments could be what I'm looking for. Anyways, I'm keeping an eye on this tool. Not sure if it really fits into my workflow, but maybe it will :)
It should indeed work for HN posts (this was something we specifically wanted). For example, here's this thread at the time of indexing: https://pastebin.com/shx8miZk
You can also access all this yourself within the datasource_browsing_history_fts table of ~/.config/persistory/persistory.db (so named for legacy reasons...)
Regarding the point about accessing the content directly, this is something we're considering. In our testing there's a bit of tension between ideal text for searching (limited formatting) and ideal text for rendering (probably markdown). However, having the local db serve as your own personal archive.org is one of our secondary goals.
Edit: added a bit more context
We do keep a fairly complete copy of the URL’s contents in the db, though we don’t currently preserve all the html or store embedded content such as images, so some formatting may be lost. Adding a method to view the stored version from the GUI is definitely on the todo list though, we want that feature too!
Being flexible and able to fit into a lot of workflows is high priority for us, if you have more thoughts on how it could more easily fit into your workflow please feel free to let us know here or on the discord
Is the idea to crawl locally? How do you crawl - sequentially or in parallel?
For the full text portion we send out a request to the URL for the full HTML, distill it (similar to "reader" mode in some browsers) and add that distilled content to the full text index.
The full-text generation happens with some concurrency but we intentionally didn't want to spam anyone's server so it's limited and takes a while to populate.