Comment by nathcd | Hacker News Reader

Comment by nathcd | Hacker News Reader

Parent Full thread

Neat! Looks like this generally works by building a static index and publishing it to IPFS [1], then having some client-side JS do a lookup in the index, and getting a metadata file for each result [2]?

I made something kinda similar a while ago [3], where I gathered a bunch of metadata about files on IPFS at the time by scraping a Searx instance, with a query like "site:ipfs.io/ipfs" or something like that. It was a quick hack job but was fun, and it's cool to see something similar on a bigger scale!

[1] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index...

[2] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index...

[3] https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKt..., https://github.com/doesntgolf/ipfs-search, https://github.com/doesntgolf/ipfs-searx-scraper

Yeah! Every question of "how would X work on IPFS?" boils down to data structures.

An example of slightly different approach (a sharded index) is the wikipedia-on-ipfs experiment from last year: https://github.com/magik6k/distributed-wiki-search && https://github.com/ipfs/distributed-wikipedia-mirror

Wow, I never thought there is something like Lunr.js! In my free time, I have been working on https://ipfsearch.xyz, thinking about pivoting into a direction of search in JS, but never getting the time to do it.

Now I'm pretty sad that something like ipfsearch (most of my work was rebuilding Lunr.js) already exists.

I guess I should do more research before embarking on a side project :( Anyway, I definitely learned a lot about how search engines work...

P.S.: now, I'm noticing that Lunr can't fetch the index over the network. OK, so not all time has been lost!

Yeah, and I think it actually uses a pretty old (0.x) version of lunr. You could definitely grab data over the network and then feed it into lunr's indexer, you'd just have to do it manually. I'd prefer to do it more like Cloudflare's stackexchange search (build the index locally, and publish the index to IPFS), rather than having the client have to fetch the data and build the index itself.

Your tool looks cool! If I find the time, maybe I'll try to dig in a bit and see if I can help with anything!

Reply on news.ycombinator.com