Is there any library/framework that helps build p2p things? That could help the ecosystem the same way web frameworks have helped build websites faster and more secure than ever.
It will be for Linux, Mac, iOS, Android, and eventually Windows.
https://www.zerotier.com/product-netcon.shtml
Currently GPL with dual licensing for proprietary closed use, which may or may not change later depending on how other things go.
Linux and iOS are working. Android and Mac next.
This gives you totally easy p2p transport but not things like a distributed database or identity, etc. But there are lots of other projects for that that become easier to use when you have transparent p2p.
Telehash: http://telehash.org/
Provides a p2p-capable messaging stack. I've worked with an earlier version (v2) for Toc Messenger [1]. V2 included an integrated DHT for public peer discovery along with the p2p messaging stack. Was actually surprisingly robust and reliable considering how young the project was at the time.
V3 seems to have pivoted towards the private mesh networking side, and spun out the DHT into a separate project, but V3 itself doesn't seem to have a clear story for integrating with an external, public DHT the last time I checked, so the intended use cases aren't quite the same as V2.
[1] http://toc.im/
Matrix: http://matrix.org/
Doesn't allow for fully peer-to-peer messaging, but is decentralized and federated like XMPP. Probably the closest thing on this list to a framework (handles both messaging and storage).
Last time I looked at it, my main concern was how much implicit trust users needed to place in homeserver providers (messages and user profiles are stored in plaintext). Unless this has changed, I don't find it viable for any app that values user privacy. Even federated networks eventually gravitate towards a large degree of centralization once the tech becomes mainstream (see email). If any servers are used at all in a privacy-minded app, they need to be zero-knowledge as far as I'm concerned, for actual user data at the very least, if not metadata as well (I realize the latter is an unsolved problem).
RemoteStorage: https://remotestorage.io/
A decentralized, federated protocol and library for data & file storage. Also used this for Toc. Worked as advertised but I'm looking to explore fully p2p alternatives for my next project.
The main problems I ran into (asides from the non-fully-p2p architecture) was the rather clunky API (for instance, it requires developers to provide a schema to fully specify objects they want to store, but doesn't provide a versioning and migration mechanism to take advantage of the schema specs, although it is planned), and the fact that encryption for data at rest wasn't implemented (until I finished working on a custom encryption layer for Toc built on top of the library).
Kinto: http://kinto.readthedocs.org/en/latest/
Similar to RemoteStorage, but decentralized discovery is planned and not implemented yet. It's built by Mozilla and apparently used in Firefox and Firefox OS (I'd assume for Firefox Sync?).
They also have a comparison table with RemoteStorage as a part of their docs if you're interested in a more detailed (but possibly impartial) comparison [2].
[2] http://kinto.readthedocs.org/en/latest/overview.html#compari...
Swarm: https://github.com/gritzko/swarm
A decentralized data replication library that's built on ops-based CRDTs [3]. Some very impressive work here. Although I recently found out that V1 of the library no longer supports fully p2p replication due to architectural changes, which made it a lot less interesting for my use cases.
[3] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...
Replikativ: https://github.com/replikativ/replikativ
The one I'm most excited about personally. Similar to Swarm, a CRDT-based data replication library, but even more ambitious (they envision an open data exchange whereby users fully control their own data, and can specify any number of applications to have access to that data, instead of the status quo where user data is kept in closed, per-application silos). There's a comparison with other alternatives, including Swarm, in their README [4].
They also plan to have pure browser-based p2p replication through WebRTC, which is exactly what I'm looking for.
It's built in and for Clojure and ClojureScript, but JavaScript support is in the works. As if I needed any more incentive to get started on my first ClojureScript app. =)
[4] https://github.com/replikativ/replikativ#alternatives
GunJS: http://gun.js.org/#step1
A decentralized, replicated graph database. The pitch is very impressive sounding, but I'm hesitant to actually use this because they promise a lot of replication magic, but the replication is built on a home-grown algorithm rather than something rigorously peer-reviewed and implemented industry-wide like CRDTs.
I'd love to hear from anyone who has actual working experience with it though.
IPFS: https://ipfs.io/
IPFS and its suite of associated libraries has already been mentioned elsewhere in this thread, so I won't elaborate on it.
I think today I would give IPFS a try.
First, Aphyr (Kyle Kingsbury of Call Me Maybe fame, Jepsen database testing suite) did some tweets about GUN ( https://twitter.com/aphyr/status/646302398575587332?ref_src=... ), and I met with him at a conference in Berlin where we both were talking. He had some concerns (wanted the conflict resolution to be able to be plug-in-play with your-own-CRDT, and if journaling is disabled then some stale/old updates would be dropped), but overall seemed positive about GUN (check his own tweets). I hope to have him do a more serious review at some point, but I also hope that his initial thoughts provides some initial industry peer review.
Tailing the first point, part of the reason why we didn't receive a more critical response from him is because we make NO claims of serializability or linearizability, which is what most academic research focuses on. In fact, GUN takes a somewhat controversial view with regards to classical things like Strong Consistency. For more information on how GUN's algorithm works, check out this recent podcast we did with the LondonJS community - https://youtu.be/qJNDplwJ8aQ?t=32m45s .
Second, GUN does use CRDTs. All CRDT means is "Convergent Replicated Data Type", which is a fancy way of saying that you'll get the same end result regardless of the ordering of the operations involved. Because GUN is offline-first it is incredibly important that it can retry an update multiple times to handle network partitions - this fundamentally results in operations not having a consistent ordering, especially when dealing with multiple peers updating (offline) simultaneously. If you watch the "Holy Grail Demo" (1min long, https://youtu.be/-i-11T5ZI9o ) you see that this does not wind up being a problem.
Third, there are other CRDTs that can be implemented on top of GUN's core. A specific instance of where GUN's default will fail you is in commutative operations (something Kyle was also concerned about), by default you cannot do increment (adding numbers) operations with GUN - partly because GUN treats a single value as being atomic. If you have a HN upvote count and you naively were to use GUN's out of the box CRDT, and two people locally added 1 to their local value at the same time... they would both converge to the same end result (5 + 1 = 6) but the intended result is 7. This intention-preserving CRDT is called a Commutative Replicated Data Type, which a specific variant on the general category. However, this does not mean that it is impossible to do this with GUN - you can, because GUN's underlying CRDT is designed to be emergent, allowing for other CRDTs to be built on top. At some point we'll (or somebody in the community) will provide these as plug-in-play module/extensions to GUN, so that way you won't have to build them yourself.
Fourth, "home-grown" is somewhat true, but I think that is giving me too much credit. GUN's underlying conflict resolution algorithm is based on a hybrid lexical, vector, and timestamp clock - hardly magic at all, in fact they are somewhat naive intentionally. The nice thing is that they "just work" for the majority of web based apps right out of the box, which is probably why it seems magical. However, they won't work (alone) for more complicated business logic, and that is why we encourage the UNIX/NodeJS philosophy of adding those algorithms on top. It is important that your bread-and-butter machine is not based on a monolithic black box (like a lot of databases are), instead you should understand the base of the system and then plug your business-specific needs on top.
Fifth, we're looking for more academic and peer review! I've already had numerous discussions with a Distributed Systems PhD at Carnegie Mellon that got poached by Uber's self driving car division, he's trying to help me find some people to put together a paper. That said, if you know anybody or could help connect me into that world it would be greatly appreciated.
Thanks so much for including us in the list, please shout if you have any more questions! :)
One question still on my mind after quickly skimming your docs: Can Gun sync directly between browsers (using something like WebRTC) without involving any servers? If so, could you point me to where it might be documented? If not, please consider this a feature request. =)
As you know though, WebRTC unfortunately requires some initial servers (STUN/ICE, or a signaling server, etc.) so it is not fool-proof P2P sadly.
We're approaching our 0.3 release, which will provide the minimum viable framework for developing fully peer-to-peer applications. There is much work to be done with regards to documentation and design, so we'd love to welcome anyone who wants to help to our community.
it's a gossip based network / database.
Check out our first active app : http://github.com/ssbc/patchwork