I built an IPFS crawler with libp2p | Hacker News Reader

I built an IPFS crawler with libp2p

adlrocha.substack.com

116 pointsadlrocha19 comments

Was hoping there was a link to the results - either a search box or some statistics. But hey - free code. Not complaining.

Also, if you haven't checked out Beaker Browser now that it's nearing 1.0 - great things are happening. This hasn't been broadcast much - but Beaker now has a built-in editor pane and forking mechanism. It's the best shot I've seen at a 'view source' 2.0.

There's been some healthy competition lately forking and editing the default twitter/mastodon clone. (Which is here: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ But forks of it are here: hyper://0e0d1685c3575cf1df94191e14b4ecf068c43bcefee787ffbbed3d1f4666cab4/ and here: hyper://b798b86cc282ae065697f36a2a07072bfa28e5fdfa1d9a8c6fac0f07ff7d6676/.) If you want to learn Javascript, this seems like an amazing way. (Kind of like jsfiddle but for actual apps.)

Is beaker going to support IPFS?

They used to, but they removed support. It makes sense to focus on just one protocol I think since they're such a lean operation. Also the guy running Beaker seems to be sort of anti-blockchain (describing himself as an "Unblockchain consultant") which might make them biased against it.

Also, IPFS urls change when the content they're pointing to changes, which makes them bad for decentralized web stuff. They are working on IPNS to handle this but I don't know what the status is.

Actually, content-based addresses is really critical for decentralized web stuff - since you can't trust a single source to name data, and instead want many different parties to participate in the same global address space in a secure and trustless manner. Check out the dweb primer for why working with a base-layer built on content addressing is so powerful: https://flyingzumwalt.gitbooks.io/decentralized-web-primer/a...

IPNS is used by many groups to create a mutable layer over IPFS (it recently got much faster in our 0.5.0 release in April), but you can also use tools like Textile, OrbitDB, or other mutable databases built on IPFS.

I read that document and it seems to me like everything in there applies to Dat also, which isn't really content-addressed. But if IPNS is fast now it seems like it could be a good replacement.

IPNS has been working for a few years already. Both the dnslink and actual IPNS. However an instance of the IPFS daemon can only serve a single IPNS address.

The real problem with IPFS is that hosting a node requires a lot of bandwidth. I regularly tried to self-host an IPFSite but it makes make internet access literally unusable. So I abandoned that idea for good and now I use a free external service for that.

Also the guy running Beaker seems to be sort of anti-blockchain (describing himself as an "Unblockchain consultant")

That's funny. I haven't checked them out, but I imagine its more the absurd hype (similar to AI) that annoys them rather than the technology itself.

Is IPFS blockchain based?

Nope - but used by lots of blockchains for things like off-chain storage (since having a content-addressable handle for a file that can never change is super critical for things like smart contracts)

> which might make them biased against it.

How is IPFS related to blockchain? On a technical level that is.

AFAIK IPFS is somewhat related to FileCoin which runs on "a blockchain". But not IPFS.

Or am I missing something?

Correct, IPFS is not a blockchain - while both are merkle-dags ('directed acyclic graphs' of hash-linked data), blockchains have all nodes sync the state collectively, while IPFS instead shards state across the entire network (such that each node maintains its own filestore).

hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac21 has been giving me a 500

Somehow your link is truncated. The complete link is: hyper://a8e9bd0f4df60ed5246a1b1f53d51a1feaeb1315266f769ac218436f12fda830/ (with a '30' at the end)

My bad

For anyone looking at https://arxiv.org/pdf/2002.07747.pdf to get an estimate on the actual # of IPFS nodes, note that these tools are trying to crawl the number of nodes acting as "DHT Servers" in the Public IPFS DHT. Nodes behind home firewalls or NATs don't become DHT routers because they aren't dialable, and other large users intentionally set many of their nodes into a lower-bandwidth client mode. Also, many applications run their own private IPFS networks (like OpenBazaar for instance) and don't participate in the Public IPFS DHT, so wouldn't be reached through a crawler here.

How is the adoption of IPFS going, any example application that are using it and are themselves being used? I'm thinking of integrading it into a product, it has the best blockchain like properties without the mining madness.

Going well! Check out all the awesome applications using IPFS in our editable ecosystem diagram: https://www.figma.com/file/2bVNnjoOmxh4TtID9kzqEG/IPFS-Appli...

A few highlights are Fleek, Textile, Audius, 3box, Anytype, Qri, Berty, and MagicLeap! Specifically in the blockchain space, there's Peepeth, Augur, Uniswap, Civic, Arbore, and Aragon.

So this is crawling the nodes not the content?

Seems so.

There is https://ipfs-search.com/ [https://github.com/ipfs-search/ipfs-search].

Reply on news.ycombinator.com