Hacker News Reader

Comment by JsonDemWitOster | Hacker News Reader

While a lot of people are concerned with local model performance, I wonder how feasible is it now to run a local indexed web search? Surely running an old school Google is possible with the beefy AI rigs today. I know the problem will be crawling which would be bottlenecked by the ISP but I use Google to search SO, Wikipedia, programming language docs, Github issues, and AWS docs. I think a feasible workflow would be to build a set of sites of most interest to you and then prioritize those in crawling.

While typing this out I remembered https://en.wikipedia.org/wiki/Google_Search_Appliance which I never personally used but shows feasibility for the idea. I'm pretty sure one of the newly-announced Macbooks is more than up to the task of matching GSA's offering.

View on HN

Impossible. The majority of websites firewall automated crawler traffic (because of the rise of the bots), only making exceptions for the largest search engines. There is no possibility of starting a new crawler.

It would be interesting to see a decentralised, residential collective that builds and publishes an index. There are surely enough interested people on HN alone that would be willing to run software at home to scrape a small slice of the internet.

Something like YaCy?

Just like that! I will give it a try

residential proxies are one of the leading causes of what's killing the internet right now

The majority of websites try to do that but they do not catch as much traffic as they think they do. A starting point for a scraper is to run it on your home connection in an undetectable web driver framework such as zendriver.

This was a knee jerk response to the first paragraph. They weren't talking about a general crawler, but a subset of Wikipedia, stack overflow, programming docs and github. You can download archives of all of those except github, and github could be queried using the api or GH cli

Most of the stuff should be bypassable with browser automation but you'd need more compute to run a full browser versus a basic uri fetch.

I imagine it'd take quite a bit of shape for the index and be hard to keep it up to date unless you restrict what it indexes.

SearXNG configured as in the OpenWebUI docs is pretty cool. My "Hello World" with a new agent framework is teaching it to use SearXNG. Hook this in as a tool and the agent can answer a lot of questions.

SearXNG is more of a metasearch, the dude who wrote it pops in on here and is working on a cool sounding project that is more like a local personal search engine, I forget the name, but I've been meaning to check it out.

There is also Common Crawl.

https://docs.openwebui.com/features/chat-conversations/web-s...

https://github.com/brian-learns/xng-agent

SearXNG has been working pretty well for me. I had an agent write the MCP then do a couple passes comparing to server side LLM web tools and exa and tweaking and it works pretty well. I also added scrapling for fetch which covers pretty much everything but sometimes is a bit context heavy.

He's working on Hister now. I really like it.

Wayback Machine full archive is like less than 50PB. Let's say you could strip multimedia and remove every patterned data to compress that into about a petabyte. The per-bit cheapest disk right now is consumer Seagate 24TB(SI; 21.8TiB usable) at ~$500, or around $1200k for just the disks.

Doable if you had couple million dollars to burn. Cheaper than private jets new.

Seems like the most practical path is some kind of distributed peer to peer contraption where you can allocate some storage and optionally participate in crawling.

I imagine if set some constraints you could get index size down quite a bit but I still suspect it'd be hard keeping up with content churn.

Edit: Looks like enwiki bz2 is coming in around 46Gi which isn't too bad considering the amount of content it contains.

https://yacy.net/

It's still kind of an open problem. There are partial solutions, but not yet really an integrated one. There is Hister: https://hister.org/ that builds some sort of drive-by index of what you're browsing anyway. And there are "true p2p" solutions like YaCy: https://yacy.net/ but it takes ages to crawl the open web (and lots of storage). There's still a lot of optimization to do in this space.

For some of that, you don't need to crawl. Wikipedia offers database dumps which you can download in one go. Lots of programming docs are managed in repos, so you can clone the repo instead. Even stackoverflow seems to have a snapshot dump (https://archive.org/details/stackexchange).

Until 10 years ago, i used Dash for that. It’s still around https://kapeli.com/dash

It’s instant, works offline, auto-updates, and includes all the websites you listed, and allows for custom ones too.

What do you use now?

https://devdocs.io/

Plausible. Figure 100 GB each of search index for Stack Overflow, Wikipedia, and GitHub issues, then add a dozen more for docs of all your favorite techs. So maybe half a terabyte. Download and build updated dumps of those once every week or two, and it'd work pretty well. Impractical, but possible.

Reply on news.ycombinator.com