Last I remember, Z-Library was having an issue scaling the DHS to handle the number of files [1]. Did those issues get resolved? How is it going now? Also, is there anything being done to ensure every file has seeders?
The security was also a concern, and the scaling had issues. Pinning millions of small items got so slow it would not function. Then I ended up having concerns over hashed based addressing being easy to censor with the architecture IPFS was using (more hub & spoke than anything, given signaling and relay servers were centralized).
I could go on but I ran into so many issues I ended up implementing my own solution that did everything I wanted. Wanted to squeeze even more performance, I’ve been converting that solution to Rust.
This was a couple years ago so maybe things have changed since I used it. Last I checked, they seemed busy on Filecoin.
The idea of IPFS is great and I want to see it succeed, but I think that they got so caught up in their jargon and modularity, the project lost track of some fundamentals.
Their intentions seemed good to me they just have an ungodly amount of financing while perhaps lacking a core vision & understanding of what is at stake.
Put another way, seems more like an academic building something rather than a seasoned industry pro.
In any case, I don’t want to disparage their project. I learned a lot from their code & concepts. We are all on our own roads towards brilliance, contributing to each other in all kinds of ways.
In this case, to be blunt, give a bunch of nerds millions / billions of dollars and see how fast their objectives change. Going from being students to having material wealth unimaginable to most suddenly gets you invited to a lot of conferences / parties / inner circles. You’re doing a lot less engineering and a lot more socializing & experiencing, and suddenly those lofty ideals & nerdy curiosities may just not matter as much any more driving around in a lambo with what you’d consider super models. But making even MORE money to do cool things just might.
Very few people exist on this planet that can resist that, and it is the explanation for most people / projects / orgs that make it big losing their way. Once you have a lot of money, suddenly life without it seems impossible, and you start to compromise values to secure more of it. It is a steep downhill journey.
Most projects are unoriginal. Just a derivation slightly improved on, or a recombination of prior works in a compelling new way. Nothing wrong with that; our universe is composed of relatively few atoms combined in all kinds of special ways.
you both are correct on all assumptions. and are now discussing how corruption with money within the existing system is both undetectable/deniable and effective.
ipfs is now, for all purposes, a honeypot against dweb. or at least a way for opposition (dweb proponents) to waste valuable time. as your experiences proved.
Let me share my 2 cents as the Developer Advocate for IPFS (employed by PL).
You claim that our work is largely ineffective but what are you referring to? The IPFS ecosystem is like a cluster of several ecosystems, e.g. Libp2p, IPLD, IPFS, Multiformats.
> Lots of code and specs but nothing really usable
Some of the libraries may not be as actively maintained as others — this is the nature of open source. But to claim nothing is usable is not accurate — there's a lot of stuff that works pretty reliably.
> Code is not very original
Who cares....
> autonat is their version of stun/ice.
It's actually Autonat, DCUtR, and Circuit Relay V2 which are pretty amazing given that it's like STUN/ICE but decentralized! Anyone in the network might help you hole punch through NAT and from network measurements the success rates average around 70% (https://youtu.be/bzL7Y1wYth8?t=1091). Saying it's just "their version of STUN/ICE) takes away from the main qualitative difference of doing it without centralized coordination servers.
> "what apps use libp2p" you won't find anything because nobody uses it.
Ethereum uses libp2p GossipSub since the beacon chain launched. Berty, Satellite (https://satellite.im/), Status (https://discuss.status.im/t/waku-version-2-pitch/1776#org900...).
Either way, I appreciate your comment, we need to be doing a better job of surfacing some of the great progress being made in the IPFS/Libp2p ecosystem.
ConsenSys is a company formed by Ethereum co-founder. He has sold out to the big banks, in the opinion of some Consensys shareholders.
https://finance.yahoo.com/news/consensys-shareholders-readyi...
"Falls says he believes both MetaMask and Infura - what he calls "the crown jewels" of ConsenSys - could have been decentralized and tokenized, and that their projected use is "completely anathema" to the peer-to-peer principles of the space.
Meanwhile, a number of teams are looking to bring "institutional DeFi" - with its known counterparties and compliant custody arrangements - to the marketplace.
"Forget about the shareholders for a minute," Falls said. "Think about the consequences of the change in the influence over these infrastructure pieces.""
Out of curiosity I was looking at the evolution of IPFS hashes. They have gotten more complex. IPFS now uses base58btc exclusively. That's "btc" as in Bitcoin. Something like rhash, which has traditionally supported hashes used in peer-to-peer protocols, has no support for base58btc.
People are now trying to associate IPFS with "Web3". For example, check out this paper published a couple of weeks ago.
"Studying the workload of a fully decentralised Web3 system: IPFS"
That's blatantly wrong. IPFS supports 25 different base representations (https://github.com/multiformats/multibase/blob/master/multib...).
In fact, recently, two community members decided to implement a new base encoding with emojis for fun:
https://cid.ipfs.tech/#%F0%9F%9A%80%F0%9F%AA%90%E2%AD%90%F0%...
https://github.com/multiformats/multihash supports at the very least SHA1 SHA2-256 SHA2-512 SHA3/Keccak Blake2b-256/Blake2b-512/Blake2s-128/Blake2s-256 Blake3 and Strobe. Hashes in IPFS are being standardised through the IETF and W3C https://www.ietf.org/id/draft-multiformats-multihash-05.html.
If you need rhash, you are welcome to submit a PR! We also have a grants program you can use to be rewarded for this.
https://consensys.net/blog/press-release/consensys-raises-65...
https://www.reuters.com/technology/blockchain-firm-consensys...
https://consensys.net/blog/press-release/protocol-labs-colla...
"In CIDv0, hashes are always encoded with base58btc. Always."
https://web.archive.org/web/20200810110206/https://proto.sch...
"Converting CID versions
You can convert any CIDv0 to CIDv1, because the implicit prefixes from v0 become explicit in v1. However, because CIDv1 supports multiple codecs and multiple bases and CIDv0 does not, not all CIDv1 can be converted to CIDv0. In fact, only CIDv1 that have the following properties can be converted to CIDv0:
multibase = base58btc multicodec = dag-pb multihash-algorithm = sha2-256 multihash-length = 32 (32 bytes, equivalent to 256 bits)"
https://proto.school/anatomy-of-a-cid/06/
Thank you @2color for the correction.
> People are now trying to associate IPFS with "Web3". For example, check out this paper published a couple of weeks ago.
IPFS has been used to host dApps for a long time — eg the Uniswap frontend. This isn't a recent thing.
It is true that Base58 is named for its original implemention in bitcoin but other than the name it has nothing to do with bitcoin.
The difference from Base64: It excludes non-alphanumeric characters and visually similar characters (I, l; O, 0) to make the output more human-friendly[0].
The problem it attempts to solve: Making bitcoin addresses more human-readable. That is likely also why IPFS uses it.
ConsenSys was founded by Joe Lubin, and is the corporate entity behind MetaMask, Infura, etc. It counts MasterCard and JP Morgan as investors.
Protocol Labs was founded by Juan Benet and is the primary sponsor of IPFS and the dev team behind Filecoin. They're funded primarily from the private token sales of FIL.
Agree with the sibling comment that the focus on modularity and solving "all of the things" rather than focusing on adoption seems to be a pattern with Protocol Labs, though I'm a big fan of what they're trying to build.
ConsenSys is an entirely different company focused on Ethereum, and neither invented nor maintain IPFS. That's Protocol Labs.
Protocol Labs didn't start out of ConsenSys either — in fact, I don't believe they have too much beneficial ownership in common at all. In addition to the separate founding stories already mentioned, you can read about how Protocol Labs was initially funded by Union Square.
Protocol Labs, as you correctly identified, is not a stooge of global corporate interests.
The #1 lesson they failed to learn from Bittorrent and Tox: your DHT must be sessionless.
QUIC is not a solution to this particular problem. Although QUIC uses UDP, it still creates and manages sessions. This is incompatible with decentralized scaling. At the scale of Bittorrent and DNS you cannot manage this kind of session state without centralization.
I'm not 100% certain what you mean by the DHT being sessionless.
> At the scale of Bittorrent and DNS you cannot manage this kind of session state without centralization.
You don't have to route content via the public DHT. Depending on what kind of data you're publishing, you can use a private DHT swarm, or delegated routing with HTTP.
> This is incompatible with decentralized scaling.
The full answer to that question is way, way, way beyond the appropriate length for an HN comment.
DHTs need to accept requests from any random IP address, anywhere on the Internet. If you allocate state, you have to deal with resource exhaustion attacks. Dealing with them turns out to be way harder than anybody thinks when they first start down this path.
Cloudflare and Google can use stateful HTTP(S) because they are so massive. They can employ hordes of people to constantly tune DOS filters to block griefers before state allocation occurs, and lawyers to deal with any griefers who are as large as they are. If you're happy with that kind of centralization you shouldn't be wasting your time with DHTs; you don't need them.
Aside from resource exhaustion attacks, there are some pretty deep lessons connected to systems and architecture. The bittorrent bootstrap nodes serve some really insane number of requests/second on very modest hardware (I'm still trying to find the link that detailed this...). The Internet root zone DNS servers were, for a very long time (and perhaps still today in one or two cases) just a few good workstations at universities.
Sessionless protocols scale like crazy, on cheap hardware. But they are very hard to design correctly. If you have VC bucks, spend them on hardware and use sessionful protocols.
It would be really hard to launch that attack with a DHT that implements appropriate pruning, drop, and block rules for bad behavior. All that can feed into a personal (& autonomous) web of trust. Any references for me to read more about this?
i.e. Cloudflare. It works great. It is centralized.
> Any references for me to read more about this?
You can score member nodes and gauge bandwidth in the known network. Nodes can set their constraints accordingly and take more requests than others. Nodes that are open to edge & can handle a lot more connections can handle the bulk of onboarding requests.
This does not need to be a centralized process.
Your ideas are intriguing to me and I wish to subscribe to your newsletter!
(I really would appreciate any pointers to additional info on something that actually works...)
Until then, what I’m building unfortunately doesn’t seem to have a peer. So I will keep at it. If it gets to a point where I believe it can be useful to others, I will share it freely.
It sounds very useful already.
I’d love to see the code even if it’s not in a quite working state yet.
Please reach out if it fits your use case.
Added to my user page to avoid creating a permanent record. I’ll reply with my personal.
Hackernoon released a blog post comparing the two projects.
https://hackernoon.com/whats-the-difference-between-ipfs-and...