Ask HN: Recommendations for Peer to Peer Frameworks?

Maybe Scuttlebutt? https://www.scuttlebutt.nz/

Scuttlebutt is advertised as a social network, but in core it is an eternal distributed log. You can pass either public or private messages and assets around.

This is a great suggestion, have started looking into it. Thanks!

Scuttlebutt and Dat (https://datproject.org/) are definitely looking like good bets right now.

Also perhaps fun to note is that we're currently hacking on a Rust version of Dat. Hopefully we'll be able to bring the Dat protocol to more languages and platforms soon! https://github.com/datrs

Oh man that makes me happy. I just posted about that this morning. Hopefully I can find a way to contribute!

Look into libp2p created by the Protocol Labs team (created IPFS). We're thinking of moving to this framework internally at Ethereum.

Second vote for libp2p. It looks very mature and generic enough to build whatever off of it. It has implementations in js and go.

If you end up rolling your own, I found js to have the most building blocks (pmp and upnp libraries).

There's an implementation in Rust: https://github.com/libp2p/rust-libp2p

Can anyone point to some higher-level libraries or open-source apps built on top of libp2p?

Apart from the obvious js-ipfs and go-ipfs, check out ipfs-cluster and LivePeer. libp2p is actually meant to be pretty high-level so I'm not aware of more high-level libraries.

Actually, a few minutes later, diggan set my understanding of your question straight :) Check out peer-crdt, the ipfs/y.js bridge, ipfs-pubsub-room, and orbitdb!

>We're thinking of moving to this framework internally at Ethereum.

Why not GNUnet? Check the recently released 181 pages long dissertation on it:

https://grothoff.org/christian/habil.pdf

Link broke, here's a new one:

https://pdfs.semanticscholar.org/153d/578869863450766a870727...

Previous Hacker News discussion here: https://news.ycombinator.com/item?id=15877908

GPL license is probably a non-starter.

Why?

> The core of Ethereum includes the consensus engine, the networking code and any supporting libraries. For C++, this includes libethereum, libp2p, libdevcore, libdevcrypto, libethcore, libevm and libevmface.

> The core of Ethereum will be released under the most liberal of licenses. This reflects our desire to have Ethereum used in as many diverse environments as possible, even those which, for various reasons can require modifications or augmentations to the software which cannot be released to the public.

https://github.com/ethereum/wiki/wiki/Licensing

definitely the most promising out there

Awesome List of P2P resources:

https://github.com/kgryte/awesome-peer-to-peer

Personal bias tilts to web tech. WebRTC will require you to centrally implement all your peer broker and logic. But take a look at something like Peer5 Video CDN to appreciate scalability:

https://www.peer5.com/

Hey, Benji from Peer5 here - our product is really cool and helps you scale your video delivery but I don't believe it works in this case.

We solve the problem of distributing HLS/DASH delivery over large p2p swarms and scaling video. Doesn't sound like it could work for OP.

Thanks a lot for keeping us in mind though :)

Hey Benji. I saw you guys setting up at NAB yesterday. Would there be a good day this week to come by and talk shop?

Hey Adam, I'm not attending NAB this year - Shachar and Hadar would love to talk (shachar@peer5.com or hadar@peer5.com) in terms of technical people attending :)

https://urbit.org/docs/

The idea with urbit (amongst others, it's a big system) is for everyone to have a personal server that's easier to administrate than a linux server. It's been posted a bunch of times https://hn.algolia.com/?query=urbit&sort=byPopularity&prefix...

I've also heard about https://zeronet.io/ and https://beakerbrowser.com/

I’m curious what your goal is. Do you want the complete immutability of transactions like bitcoin/blockchain; or do you want the peer/seed capability like BitTorrent and IPFS with a DHT (Kademlia)? I’d be interested in seeing a good list of the technology choices that can make up a P2P project if anyone knows of one, please share.

But you definitely should checkout IPFS and Kademlia.

Maybe check out Dat[0] - especially useful if you need some kind of distributed data store. The main implementation is in Javascript.

[0] https://datproject.org/

I really wish there was an additional implementation in something like Rust (personal preference, but C would be good as it's fairly easy to bind to) for DAT, because what they're doing is rather cool and the experience is tidy.

That’s in the works: https://github.com/datrs

Awesome. I'm really excited for that, I'll have to try to contribute.

The 181 pages long GNUnet dissertation very recently got released: https://grothoff.org/christian/habil.pdf

I'd strongly recommend going for GNUnet, nobody takes me serious when I say this, but it WILL surplant IP eventually.

Link broken.

https://pdfs.semanticscholar.org/153d/578869863450766a870727...

Previous Hacker News discussion here: https://news.ycombinator.com/item?id=15877908

http://web.archive.org/web/20180327164208if_/https://grothof...

I agree. I think gnunet gets a lot right.

> but it WILL surplant IP eventually.

The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS.

>The painful move from IPv4 to IPv6 suggests that this is unlikely. More likely is an overlay over IP, TCP, or even HTTPS.

GNUnet can function as an overlay and an _underlay_ simultaneously, quoting from the dissertation:

> GNUnet is an overlay network, as it is initially supposed to primarily operate over the existing Internet network. However, GNUnet does not assume that this is always the case. For example, GNUnet can also operate directly over WLAN or Bluetooth to create an ad-hoc wireless mesh network, and with the GNUnet “PT/VPN” subsystems we can run TCP/IP over GNUnet. So in this configuration, GNUnet would be an underlay network, a bit like MPLS or B.A.T.M.A.N. [NAL07]. In reality, we in fact expect to see both at the same time: some peers will run over the existing Internet, while others may connect to GNUnet on Layer 2.

The WebTorrent Project [1] has a whole suite of tools for decentralized applications based around BitTorrent. It's not a framework per-se, but it breaks up the protocol into libraries for your use.

If you're interested in Ethereum, it has other non-blockchain technologies that you can use for your apps. It has the Swarm network for storing bits of data, the Whisper network for chat and messaging, as well as Public-Key signing to verify messages. I'm using the latter functionality for a demo to replace password as the security mechanism.

[1]: https://github.com/webtorrent

Secure Scuttlebutt. It's probably the only one seeing everyday use to get things done. Patchwork, the flagship app is as good as Facebook/Twitter. Scuttlers often use git over SSB, even.

What SSB gets right and the others don't, is the incentive to mirror or replicate. Virtual currency is a weak incentive in an early stage network - but friendship isn't; you mirror your friends' data.

The basic API to program the dapp can be seen here - http://scuttlebot.io/ But for more advanced uses see https://youtu.be/f_baWUW4R8Y

Maybe checkout the dat project https://datproject.org/ which is whats used by BeakerBroswer etc.

Kademlia DHT:

https://en.wikipedia.org/wiki/Kademlia

https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia...

How about IPFS? OpenBazaar is an IPFS case study. They recently released a revamped v2 built with Go.

That's interesting, I hadn't thought about IPFS - it doesn't really have a strong message passing semantic so much as it has good replication properties, but maybe you could build a distributed log on top of it.

By good "replication properties" you mean "having to run many scripts/nodes yourself" to get widely replicated that is true.

We found IPFS in general to be awesome software that just wasn't quite ready for production.

There's IPFS-cluster[0] but that's still in capital-letters ALPHA.

https://github.com/ipfs/ipfs-cluster

They are working on this: https://github.com/orbitdb/ipfs-log

Yeah, I'd look at IPFS's networking library, libp2p. It's the peer-to-peer backbone of IPFS and has a big community and active development https://libp2p.io/. It's also implemented in both go and js (as most of IPFS is)

The wire protocol is practically almost not documented.

We moved away from libp2p at the project I work on because of major instability problems in browsers. Our dapp that used libp2p would crash after a few minutes and was generally not easy to use.

EDIT: I'm referring to the JS version here.

Sorry to hear you had move to something else -- my guess is these crashes were from WebRTC, which isn't very performant in browsers yet. Especially with more than a handful of WebRTC connections, we've seen it crash regularly.

We've switched the default browser-to-browser transport in js-libp2p to a websockets relay recently, so you should give it another try.

Did you try to investigate the stack trace or file a bug upstream? No snark intended!

Yes, of course, and it was WebRTC. We ended up with a better solution (encoding the metadata we were storing in IPFS in a bytes32 stored in the Ethereum blockchain).

There were extra scripts and setup. Overall moving off simplified our stack, so it was probably a good move despite libp2p's crashing.

Could be totally fixed by now for all I know.

That said, it was believed that most Protocol Labs resources were/would be devoted to Filecoin, which is probably a safe bet.

We have no intention of letting libp2p stagnate - blog post with announcements coming soon!

I'd love to read technical details about what they built with it and how they manage replication. Will look for some and repost if I find.

Look at the IPFS Pubsub and Y.js integration, that e.g. PeerPad uses to build distributed text editing: https://github.com/ipfs-shipyard/peer-pad/blob/master/docs/A...

Another project you might find useful, that we at https://www.maidsafe.net are actively working on, is https://github.com/maidsafe/crust/.

It's a generic peer-to-peer networking library written in Rust.

  * it connects two peers together
  * various NAT traversal techniques are implemented: hole punching, UPnP/IGD, etc.
  * it supports both TCP and UDP. In case of UDP it uses uTP: http://www.bittorrent.org/beps/bep_0029.html
  * all messages are encrypted
  * it implements automatic peer discovery on LAN
  * it caches it's previous connections and is able to reuse them in the future
  * etc.

The virtual synchrony protocols don't get enough love. Conceptually, they're sort of like if you performed PAXOS to determine ordering for a distributed finite state machine. Let's you build all sorts of distributed applications where you trust the nodes.

https://en.wikipedia.org/wiki/Virtual_synchrony

http://www.jgroups.org/

https://archive.codeplex.com/?p=vsync

I am using https://hypelabs.io/

Pretty good for small payloads.

Re blockchain technologies: while there is an obvious intersection between the world of blockchain and P2P, blockchain technologies are slow because BFT adds many requirements to the game. Basically blockchain technologies use P2P to reach a common state and this could not be necessary in your use case.

You can take a look at OpenBazaar and Particl for P2P implementations.

ZeroMQ http://zguide.zeromq.org/py:all ... "In One Hundred Words: ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ZeroMQ is from iMatix and is LGPLv3 open source."

Peer Name Resolution Protocol (PNRP)

- Developed by Microsoft

- Open Specification

- DHT-based

- IPv6 only

- Windows support since Windows XP SP 3

- .NET support since .NET Framework 3.5 (System.Net.PeerToPeer)

[1]: https://en.wikipedia.org/wiki/Peer_Name_Resolution_Protocol

[2]: https://msdn.microsoft.com/en-us/library/aa371699.aspx "PNRP Namespace Provider API"

[3]: https://msdn.microsoft.com/en-us/library/bb726971.aspx "Peer Name Resolution Protocol"

Microsoft had a few cool things, but it all seemed .net specific. Check out this in WCF https://msdn.microsoft.com/en-us/library/cc297274.aspx

Blockstack looks like it would fit here:

https://github.com/blockstack/blockstack/blob/master/README....

+1 for Blockstack

If you plan on having any collaborative editing features, Yjs [1] is a p2p framework utilizing CRDTs with support for IPFS.

[1] https://github.com/y-js/yjs

I recently discovered zyre. Its from the same guy that brought us the wonderful zeromq library

While it does a lot of the connection and communication part, it lacks a way to bypass nat routers.

Here is a book on zyre http://zguide.zeromq.org/php:chapter8

I also found this thing called pwnat which bypasses nat routers without the need for UDP hole punching !

https://github.com/samyk/pwnat

I feel like if someone made a high level wrapper combining these two, we can have a very good framework.

> supports seeds and peer discovery, message passing, maybe firewall / UPNP support, etc.

Our audacious 'IPv8' work is now deployed live. Might fit you needs. Includes above features in a low-complexity approach. https://github.com/Tribler/py-ipv8 Our fresh IETF Internet Standard draft of this work: https://tools.ietf.org/html/draft-pouwelse-trustchain-00

TomP2P (https://tomp2p.net/) is a library handling firewall / UPNP support, however, the focus is academic/experimental.

Worth having a look into the Holochain framework:

https://holochain.org/

Depending on the application(s) you have in mind.

Holochain is interesting for having a different approach then the blockchain-based frameworks, which are inefficient

How about something [completely different](http://firestr.com)

Maki 0.3 will include components from Fabric: https://fabric.fm — automatically available to anyone upgrading from 0.2. :)

I2P works well enough.

Check this great talk: https://youtu.be/izQFMADw70w

WebTorrent.

Look into crdt it’s the shit for collaborative editing (idk if that’s what you ares looking for tho).

Do you happen to know of a decent and easy-ish crdt library in Go?

Check github there’s plenty.

CRDT is definitely a good pattern for the data passing itself, but one of the really tough parts of these distributed apps is the plumbing work, just getting peers to reliability communicate. That's part of what I'm looking for.