How IPFS is broken | Hacker News Reader

How IPFS is broken

fiatjaf.com

59 pointsharporoeder21 comments

I can't say I disagree with any of this (except maybe the cryptocurrency bit), this was my experience as well. I even made a pinning service (https://eternum.io) years ago, but I shut it down after having to deal with the umpteenth frustration of the IPFS server being slow, not discovering data, not pinning, taking up all the resources, taking up all the space, not pinning, failing to find other nodes, and not pinning.

I think IPFS is a great idea, but I don't think IPFS is a good IPFS. Give me something that is a content-addressable network of people who want to archive/store sets, that sounds like a much better thing.

Imagine: You have 200 GB of free disk space and you want to donate it to the Internet Archive. You connect to its tracker and say "give me 200 GB of your rarest content". The tracker obliges, and soon you have 200 GB of blocks. Or, you can ask people to help you keep your site online, so they pin up to X GB of it. Or, you dedicate Y GB to your OS's packages, and you can fetch them from and send them to other people without needing to contact the package servers at all.

This sounds very much like "BitTorrent but with extensible data sets". Maybe I'll see if it's close enough that I can build it without too much hassle, hmm...

> give me 200 GB of your rarest content

This sounds like a great utility indeed, I'd love it too !, over time it would make the overall network more and more resilient :D

I don't know if it would be one network, though. You'd definitely need to be able to look up data, obviously, so you could talk to the network to do that, but I think trackers would be a good way to solve some of the issues of IPFS, which means that maybe each tracker had their own "network".

I've wanted something like this many times, though, where I could help keep the IA online. If I could tell my site's visitors "help keep this site online, please donate 1 GB", it would be great.

So https://collab.ipfscluster.io/ shows how you can essentially have content distributed to followers, that may come and go, fetch updates as they will.

This is BitTorrent with RSS. I built a proof of concept for Libgen and the Internet Archive. The RSS feed spits out torrents least to most seeded, the BitTorrent client joins torrent swarms advertised in the feed until disk space limits are reached. Env vars injected into the Docker container are the RSS feed url and max disk space to use in GB. Very straightforward.

I too was disappointed with IPFS, and my goal was to build the equivalent of the ArchiveTeam Warrior, but for rapid mirroring of existing sets versus archiving ops.

That would work, but I don't like the fact that torrents are separate. I'd like the entire dataset to be one thing. Maybe I'm reinventing Dat?

While I don’t know if IPFA solves this properly, but I’d like a situation where I can move around and organise my content while it still being available to the torrent client.

If I have an arxiv backup, I can’t re-use the same for a arxiv-backup-torrent easily.

I get that disk space is cheap, but I see the allure of a content addressable system to solve for this.

this can be solved on the file system layer below.

on my laptop with btrfs this requires no special effort: simply copying the files and renaming them suffices. nautilus (gnome file manager) uses copy on write, and coreutils "cp" also defaults to "reflink" these days.

Doesn't work for torrents with RAR/ZIP files. The ref-link is also one-way. If I have existing content on my disk, and want to (partially) seed a torrent - can't do that easily.

I'd like a way to say: "Everything inside /public is public-domain content, seed it against any valid torrent if someone shows up with a valid hash"

This was written a while ago (2019-2020) and I would say most of the complains are at least innacurate today.

While some of the criticism was valid (and many things were fixed), I find the tone and the personal attacks in linked pages a bit over the top (not sure what's the point?). I can only smile at some fancy misconceptions and completely wrong predictions.

Some things are still missing: NAT hole punching (without central servers!) and indexing supernodes are things that are literally landing these days and will further improve content routing and providing in the network.

IPFS has many moving pieces so it can be disorienting and frustrating, but its also really cool to understand it and see it work.

Disclaimer: I work on some IPFS things.

What do you think the biggest disparities are in the page? I've been considering trying to host an IPFS service for a while now.

The DHT had a big upgrade. Nodes detect their own connectivity status and configure themselves accordingly so that they are not a burden to the network when they cannot be contacted. This has improved DHT lookups significantly, along with publishing, IPNS etc.

I know it is possible to run a public IPFS gateway on 1VCPU+2GB of RAM without issues (+ nginx). So much for the need for a "really powerful computer". In fact, it is more difficult to configure IPFS to take advantage of a really powerful computer than doing it for a low-spec machine (defaults are very conservative).

An IPFS peer does not drop connections to peers you have manually connected to. Bitswap (the data interchange protocol) has overhead, true, but to say it is the slowest or that things are stuck for days is a big exaggeration.

The author complains about databases on IPFS but I'm not sure about what exactly. Pubsub is a very good way to notify changes to content and it works at scale (see Filecoin). In fact, IPNS also works over pubsub, not only DHT.

The author also bashes on "IPFS has used different kinds of hashing algorithms" which is a lie as it always used sha256 by default. A CID obtained 5 years ago works today. In fact, the default options to add content are exactly the same (256KB-blocks, CIDv0, balanced-DAG). In any case, no content-address has ever been broken on upgrade.

I'm not sure about the obsession about "flirtation with Ethereum". I would seem the author hates "the Ethereum hoax" and prefers Bitcoin. shrug.

In the end, p2p is pretty difficult and IPFS solves many things only to find new issues as the network heavily grows, which is frustrating, but IPFS is nevertheless in constant evolution and improvement.

I don't know if that helps really. All you do in your comment is reiterate the issues and opine that they are not that bad. But they were bad enough for those several times when I tried to use ipfs for my projects, and I am not a noob. There isn't any way some average user would get a value from this project.

I think a lot comes down to misrepresentation. When your product doesn't work, make it clear that it's a pre-alpha. When I see version 5.0 released, I expect to see something not just workable, but also mature and stable.

Latest version of go-ipfs is v0.11.0, not v5.0.

In any case, IPFS is not a "product", it is a number of protocols and layers sometimes used together and sometimes not (libp2p, ipld, kad-dht, gossipsub, bitswap, graphsync...).

And all of them are in production and sucessfully used, but they also have different levels of maturity and evolve as we have the need and capacity.

The fact that the state is not the final "interplanetary distributed rainbow that replaces the web as we know it" doesn't mean it is completely broken either.

The hash will change depending on how the file is broken down into chunks and and a different hash can be used for multiple versions of the same file. I expected to be able to compute a hash and see whether the file is in the IPFS network; then upload it if it not in the network. That just feels wrong to me.

Because it is not the hash of the file, it is the hash of a DAG that contains file chunks on its leaves. And this DAG serves the purpose of making content copy+verification simpler, while also protecting the peers in the network from abuse (no chunk larger than 4MB can be transferred). Sure, people can choose if they edit the parameters that dictate how these dags are built and use non-defaults.

I understand this complain as it is not obvious why things are like they are, but there are somewhat good reasons!

> I expected to be able to compute a hash and see whether the file is in the IPFS network; then upload it if it not in the network.

There is no upload in IPFS. Either you serve the file from your peer or you don't. It should be irrelevant whether that file was already present in the network or not.

Hello, you appear to be well versed in the state of IPFS. If you don't mind, in your views, are there any stumbling blocks in the design and implementation of IPFS that could prevent it from achieving itss ambitions? Or maybe big, current challenges it's facing that might hamper its adoption? Things being worked on?

Thanks.

IPFS domaim is very wide so different issues affect different parts in different ways.

One major hurdle is browsers not being made for p2p. We will need http gateways and other centralized compatibility layers for the time being, which are costly and very prone to abuse. I have hopes in better webrtc support to improve interop with browsers.

Other than that, a DHT at the scales we aim for in terms of indexed content is a problem when 95% of nodes are clients which don't contribute to it. NAT hole punching and supernodes will help.

With IPFS becoming the delivery layer for content stored on Filecoin, and being the standard storage layer for NFTs, there are certainly incentives for the protocols to be used and usage to keep growing heavily.

The main problem with IPFS is chunking, if file is uploaded 2 times into IPFS it can result in two different hashes.

It would be really great if we could just access ipfs://[sha256_of_file]

The premise seems to be that IPFS takes too many resources because of DHT, and DHT takes too many resources because it opens too many connections.

Possibly a single setting needs to be tuned based on a discovered connection thruput? If the entire and sole argument is that is opens too many connections, well, you know, just open fewer connections. This isn't a valid complaint about BitTorrent (in which every client allows for max connection tuning and good clients tune connection making behavior dynamically), and it's not a valid complaint against IPFS in general, just the current state of the code.

There is a connection manager (that kills connections when they go over a limit), and I think there is a connection rate limiter that limits how many connections are initiated. Some connections are protected (those manually initiated, or those that are being used to transfer some data etc).

It is however true heavier IPFS usage results in more resources used, or slower experience in general. It is also true that the connection manager could be improved to be more expedient in handling spikes.

Reply on news.ycombinator.com