Hacker News Reader

Comment by nostrademons | Hacker News Reader

I love how people are rediscovering the previous generation of P2P software from 15-20 years ago. Reading through these papers was a huge part of my college procrastination; maybe now that cryptocurrencies give a way to provide economic incentives to those who contribute computing power, we might see a renaissance of some of these older ideas.

View on HN

Interestingly, the original Kademlia paper did not make it into any of the big p2p conferences at the time. It was published at some 2nd tier p2p workshop. It was only when the system gained actual adoption, that people referenced the paper.

In contrast, Chord, which was published 1 year earlier in 2001 is one of the most cited papers in computer science. However, Chord was not practical. It assumed the availability of bi-directional connections between any two hosts. However, NATs had just started to appear, which broke Chord. Kademlia, which does not require peers to maintain a well-defined routing structure (the ring in Chord) can muddle through with NATs - because the routing table is a huge messy tree and if any branches lead you to the destination, it will kind of work.

Being the engine behind trackerless BitTorrent gave it quite some exposure too. I do not know if it was before or after it became known in academia.

My point was that it became the engine behind trackerless BT because it worked in the presence of NATs - even if other architectures, like Chord, had superior properties (if you assume an unrealistic model of the open internet).

The other point of note is that Kademlia was not designed to work in the presence of NATs. That was a feature whose value was only appreciated when ISPs ran out of open IP addresses and NAT'd up whole networks. It shows the value of exploratory, basic research.

It is a key part of the implementation of Holochain (https://holo.host).

It turns out that global consensus is not required to maintain accounting balance in a global-scale ledger. DHTs and their validation of the data they host is a big part of maintaining the invariant of the system.

I have been excited about this project for quite some time. They recently released their Rust client, although without networking. I cannot wait to play with it further as it holds the most promise amongst all "Web3" decentalized platforms.

Kademlia is still in widespread use today. For example Bittorrent uses it to resolve magnet links.

Quite handy for any program to find a copy of itself running anywhere on the internet. It's the best solution I've found for peer discovery that doesn't depend on a central host.

There's a few implementations for most platforms. At least Java, Javascript, C++, C, and Go anyways.

Doesn't Kad require an initial bootstrap node? How do you find it?

Yes but that doesn't require a central host. If your friend is connected to the network you can join it via them.

If I've just downloaded SomeCoolNewTool that's using the network, are there bootstrap hosts pre-loaded? Who's my "friend" in that case? Are there trust issues here?

In practice, SomeCoolNewTool will be bundled with some seed hosts.

Generally if you downloaded it from somewhere/someone you can get a working node from them. Often binaries have a few boostrap hosts hardwired, but then save state once launched. Typically a host tracks several hosts per bucket, and a bucket for each log(n) of the population. So with a million hosts your client will track log(million)*4 or similar. Those are usually written to disk when you close, so you can try those when you launch next.

Clients often broadcast on the local lan to find peers that way.

Also other methods are common, for instance bittorrent clients often use the PEX standard to exchange peers, those peers are often on the same DHT, although I think there might be 2 distinct DHT networks among the popular bittorrent clients.

Worst case you could start randomly walking the IPv4 space, given that there's millions of active clients it wouldn't be long before you found one. Likely only take you a few 1000 random IPs before you found someone on the DHT.

IPFS is using kademlia, and iirc zeronet are still working on it. I don't think there currently are many alternatives actually being used. Haven't seen many projects using chord or pastry these days.

Ethereum also uses Kademlia for its DevP2P mechanism. We recently implemented this in Exthereum/Mana.

I still miss the first Napster and later the OpenNap network with its OSS clients such as Lopster. Way flawed and not scalable at all compared to BitTorrent, but it allowed 1 to 1 connections between people, so that one day I could browse a random guy shared hard drive because he had some interesting music, see a band named "Spock's Beard" and being a trekkie myself, albeit a moderate one, I simply had to hear what they sounded like, and after an eternity due to then slow connections, thanks to other bands discovered the same way, I realized that progressive rock was (and still is) even more alive today than in the 70s, although obviously not mainstream. So a huge thankyou to the old flawed and slow Napster as well.

Kademlia is also an interesting option for a small and resilient key-value store between a handful of hosts.

DHT: Distributed Hash Table

...and?

well, In theory you can store (and fetch) the data directly, rather than just using it as a routing table.

Of course, that's why I wrote "small key-value store"...

Reply on news.ycombinator.com