Show HN: Intercepting proxy for semantic search over visited pages

github.com

14 points·lynx97··4 comments

I love the general shape of this, of having some semantic record of what you browse. It's been a long goal/dream and this is an interesting neat hack to get there, a path towards that future that hasn't been open until recently.

I haven't run this yet/don't know what the UI looks like (nudge nudge), but looks like this is less for record-keeping? More for finding similar thing to what you are seeing now, although it does have a cached view?

It stores the text version of every page you visited via the proxy, and lets you do semantic search on all of them. You can retrieve the stored text in the search interface, much like the "Cached" link on old Google worked. The search UI is as minimal as it can get, likely too minimal for those used to the modern web...

The idea is to be able to search for something you've seen recently, but maybe haven't bookmarked.

btw, currently powered by pure.md

I am considering to add markitdown support, but my initial tests of markitdown gave rather subpar results. I'll wait a few weeks until the author (of markitdown) comes back from holidays... If the project continues to improve, it might be a good match. Right now, pure.md gives way better results.

I made another approach that detects browser tab text changes with a Userscript that sends changes as Markdown to a server that appends to a JSONL file. MitM might be interesting to compare: https://gist.github.com/lukestanley/c3a37ab61a45e72b74995a5c... (two files here BTW)

Sending requests to pure.md (?) seems pretty insecure and inefficient to me. If you're going to MitM yourself, KEEP IT LOCAL, then use reliable code to convert HTML to Markdown efficiently with a queue, drop text parts that are noise, summarise if that's the format desired, and THEN embed!

Reply on news.ycombinator.com