Hacker News Reader

Comment by jgrahamc | Hacker News Reader

This is my current favourite use case (from the blog https://blog.cloudflare.com/e2e-integrity/)

I wanted to provide an example of the kinds of secure, performant applications that are possible with IPFS, and this made building a search engine seem like a prime candidate. Rather than steal Protocol Labs' idea of 'Wikipedia on IPFS', we decided to take the Kiwix archives of all the different StackExchange websites and build a distributed search engine on top of that. You can play with the finished product here: https://ipfs-sec.stackexchange.cloudflare-ipfs.com

View on HN

Neat! Looks like this generally works by building a static index and publishing it to IPFS [1], then having some client-side JS do a lookup in the index, and getting a metadata file for each result [2]?

I made something kinda similar a while ago [3], where I gathered a bunch of metadata about files on IPFS at the time by scraping a Searx instance, with a query like "site:ipfs.io/ipfs" or something like that. It was a quick hack job but was fun, and it's cool to see something similar on a bigger scale!

[1] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index...

[2] like https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ai/_index...

[3] https://ipfs.io/ipfs/QmYo5ZWqNW4ib1Ck4zdm6EKteX3zZWw1j4CVfKt..., https://github.com/doesntgolf/ipfs-search, https://github.com/doesntgolf/ipfs-searx-scraper

Yeah! Every question of "how would X work on IPFS?" boils down to data structures.

An example of slightly different approach (a sharded index) is the wikipedia-on-ipfs experiment from last year: https://github.com/magik6k/distributed-wiki-search && https://github.com/ipfs/distributed-wikipedia-mirror

Wow, I never thought there is something like Lunr.js! In my free time, I have been working on https://ipfsearch.xyz, thinking about pivoting into a direction of search in JS, but never getting the time to do it.

Now I'm pretty sad that something like ipfsearch (most of my work was rebuilding Lunr.js) already exists.

I guess I should do more research before embarking on a side project :( Anyway, I definitely learned a lot about how search engines work...

P.S.: now, I'm noticing that Lunr can't fetch the index over the network. OK, so not all time has been lost!

Yeah, and I think it actually uses a pretty old (0.x) version of lunr. You could definitely grab data over the network and then feed it into lunr's indexer, you'd just have to do it manually. I'd prefer to do it more like Cloudflare's stackexchange search (build the index locally, and publish the index to IPFS), rather than having the client have to fetch the data and build the index itself.

Your tool looks cool! If I find the time, maybe I'll try to dig in a bit and see if I can help with anything!

Considering I want to use this as it is meant to be used, distributed, how do I pin all of say https://ipfs-sec.stackexchange.cloudflare-ipfs.com/ham/ (including the search) to my local IPFS node so I can use it offline?

Normally you would go about it like this:

    dig +short txt _dnslink.ipfs-sec.stackexchange.cloudflare-ipfs.com

^ will get you the root hash. Then for finding the hash of that particular directory:

    ipfs resolve /ipfs/QmVsEiBRsdiHMvmDVJTNtrXNqSKYdtoepcoZxUrQ8rU7qR/ham

But unfortunately it errors, probably because it's a HAMT sharded directory and the `resolve` call hasn't been changed to accept them yet. There is a open issue about it here: https://github.com/ipfs/go-ipfs/issues/5270

Edit: Actually found a work-around.

Loading up the hash via our IPLD Explorer reveals the (re-)name of the directory, which happens to be `05ham` and we can call `resolve` on that: "ipfs resolve /ipfs/QmVsEiBRsdiHMvmDVJTNtrXNqSKYdtoepcoZxUrQ8rU7qR/05ham"

Then we get the `/ipfs/zdj7WZpLoZYr4AoDUdpCNBVHLi1S8y6muFoCgwUa9kmxUpYCr` hash which you can use for pinning.

Now we just need a browser extension that does all that.

If pinning was a button press away it would be much more attractive than making a dns query, resolve the result and pinning that.

Indeed, that's a good idea! I took the freedom to open a issue about this idea on the repository for the IPFS-Companion extension here: https://github.com/ipfs-shipyard/ipfs-companion/issues/581

So I've been trying to get "ipfs pin add --progress /ipfs/zdj7WZpLoZYr4AoDUdpCNBVHLi1S8y6muFoCgwUa9kmxUpYCr" to complete but it just takes hours and hours. Is this normal? What might be the issue?

Does it get stuck or are you still seeing that blocks are being fetched? If it's still increasing, it's just very large. You can check the download rate by doing `ipfs stats bw --poll`.

Weird thing is, I'm not seeing anyone providing it but I can still pin it. Currently on `Fetched/Processed 1661 nodes` but I'll leave it running and see if it finishes.

But doing `ipfs dht findprovs zdj7WZpLoZYr4AoDUdpCNBVHLi1S8y6muFoCgwUa9kmxUpYCr` should list everyone who is providing it, but I'm getting no responses, which is weird.

Edit: Hm, got stuck at 2007 blocks, are you seeing the same thing?

Fetched/Processed 6077 nodes

and ~/.ipfs is not even 40 Megabytes...

How does IFPS index new content that's added to the stackexchange network daily? And/or how long does it take to add new data?

The site is a snapshot of StackExchange data. If a new snapshot gets published, a new index would be generated and uploaded with it by the uploader.

Ah. So it happens manually? Do automated backups occur? Or is that not the/a possible goal of IFPS? Just trying to understand.

The automation is something you can do relatively easily with the IPFS libraries, API, or CLI.

Ideally your approach to pulling the data from its source into IPFS is somewhat deterministic, so that you get de-duplication and generally more efficient updates.

Stack Exchange for Kiwix, yes! I can't find Stack Overflow though - has anybody tried scraping it with https://github.com/openzim/sotoki ?

Kiwix has an archive of StackOverflow but its gigantic. :( We barely got the Math StackExchange uploaded in time.

Great Scott! 122GB. A weekly update of the local rpi would seem quite irresponsible on a regular basis - how would I use rsync with IPFS? (I could not find Wayne Davison's user id on HN, he seems to maintain rsync) - Are zim files optimized for rsync?

Reply on news.ycombinator.com