PeerPad – A realtime P2P collaborative editing tool powered by IPFS | Hacker News Reader

PeerPad – A realtime P2P collaborative editing tool powered by IPFS

peerpad.net

123 pointsar024 comments

Hi everyone!

Excited that you found PeerPad so fast. We've released its alfa just a week ago at Mozfest (https://github.com/MozillaFoundation/mozfest-program-2017/is...).

I see that there are some questions about PeerPad's architecture. If you would like to know how it works, I invite you to glance through the Architecture.md at https://github.com/ipfs-shipyard/peerpad/blob/master/docs/AR... If there are things that are not clear to you please open an issue so that we can make it better.

Also, you can expect to see soon a video tutorial of how to build a p2p flipchart using the same underlying kv store that PeerPad uses, y.js + IPFS. If you can't wait you can always take a look at https://github.com/ipfs-shipyard/p2p-flipchart and see how it works :)

We will continue working on PeerPad and add the remaining features necessary to make it usable for meeting notes, publishing and more. You can track the development and also help us built it through the Github Repo https://github.com/ipfs-shipyard/peerpad

Thank you \o/

Note that https://peerpad.net is hosted right out of a go-ipfs gateway. You can instead also use https://ipfs.io/ipns/peerpad.net or http://localhost:8080/ipns/peerpad.net, or use a specific version of it:

    > open https://ipfs.io$(ipfs resolve /ipns/peerpad.net)

You should add org-mode as a document type and you would be golden. http://mooz.github.io/org-js/#header-1-1

Great idea! Created a issue for tracking it: https://github.com/ipfs-shipyard/peerpad/issues/106

Thanks :)

It didn't work for me, but might be busy.

This is IPFS and it theory will use the javascript IPFS server in your browser unless it can find a local IPFS daemon on your machine. (Or perhaps it uses one of the gateways?)

How do they prevent the data from disappearing if everyone disconnects?

Hm, that's no good. Did you run in a private tab? It also take some seconds before the node is initialized and ready to be used.

Currently, PeerPad only uses the JavaScript implementation of IPFS (https://github.com/ipfs/js-ipfs) but using a local daemon if it exists makes sense.

> How do they prevent the data from disappearing if everyone disconnects?

Right now, we're not preventing that. You'll have to be online to share the content (or have someone else pin the content).

We do have a open issue for being able to select a pinning service, which would solve this issue: https://github.com/ipfs-shipyard/peerpad/issues/90

If you don't mind me asking, what browser and version did you use?

It uses WebRTC for peer-to-peer communication, which is yet unsupported / untested in some browsers.

To answer your question: The collaborative data is saved locally by every participating peer.

There are plans to add remote tracking and pinning, increasing the persistence and availability guarantees: https://github.com/ipfs-shipyard/peerpad/issues/90

How would remote pinning work?

If I understand correctly, each keystroke in the pad generates a new snapshot, which has a distinct IPFS ID.

Should the remote pinning service publish a list of all IPFS IDs on the p2p network, and store each individual document (each one keystroke away from the previous one) in the history graph?

Should it keep the complete list of CRDT operations, and map IPFS IDs to CRDT operations IDs, so that it can reconstruct any document when it is requested?

Is there a way to use the properties of CRDT that I am not seeing?

Remote pinning would work as any other node that you give the permissions to read the feed. This node would follow the CRDT changes, persisting them locally.

Each keystroke produces a change in the CRDT, which is then eventually propagated to all participating nodes.

Each CRDT message is signed and encrypted before being sent over the IPFS pubsub network.

This protocol is not IPFS-specific, but there are plans to change this: https://github.com/ipfs-shipyard/peerpad/issues/107

The snapshotting occurs over IPFS, producing a static and encrypted self-contained snapshot, published over IPFS.

It is possible to access the CRDT properties inside the core library, where the CRDT is formed: https://github.com/ipfs-shipyard/peerpad-core/blob/master/sr... . This could be exposed if you would require it..

> didn't work

Nevermind. I was testing quickly in the airport between flights. Now I see I just had a flaky connection.

Thanks for reporting back and happy to hear it was just a flaky connection, even though we want to make IPFS make better on flaky connections.

Hey, David/others I'm very impressed with your work on Y-JS (the CRDT library powering this). I'm the author of gun, which is currently (as far as I know from stats) the most popular CRDT-based data sync (MIT/ZLIB/Apache2 licensed, 6.6K+ stars, https://github.com/amark/gun ).

Usually I'm pretty skeptical of most systems, but I'm rather particularly impressed with yours. I also do not think you are getting the recognition you deserve, because the truly novel work here is Y-JS not IPFS (even though is IPFS is great), yet Y-JS is almost just a footnote on the page which is sad.

The first couple documents (on architecture and stuff) didn't seem to explore the CRDT that is used. I'll have to follow-through on other sub-documents/pages to read more, but wanted to get your thoughts first (and BTW, thanks for all the documentation you have already provided, it is impressive, and I know how hard it is to try and get everything shipped, so the last thing I want to be is an annoying HN person whining about why you didn't already have X or Y or Z finished, and wanted to give you a chance to reply directly).

I was able to rather quickly ( https://www.dropbox.com/s/8uxzq6d47szu4w1/Screenshot%202017-... ) get the document to go out of sync, but I understand this could just be a trivial implementation detail (and often is) NOT a problem with the CRDT, but wanted to check. Given your bachelor thesis, I know you are all too aware of these things, P2P collaborative rich-text is one of the hardest problems out there in the space. Your Y-JS system seems fairly "generalizable", no? Which is unique amongst CRDTs (ours at gun is as well, so I get the nuance). If that is true, usually P2P rich-text then requires a specialized CRDT on top of the generalizable one, is this what you all did?

For instance, Martin Kleppmann did an excellent explanation of explaining theirs here: https://youtu.be/yCcWpzY8dIA?t=16m26s . I was wondering if you did something similar?

We had to implement our own, which I did one of my infamous "cartoon explainers" on here: http://gun.js.org/explainers/school/class.html . However Martin actually brings up an edge case I hadn't thought of after 4 years of analyzing the system! But thankfully he provides a patch on how to fix it. ;)

Anyways, wanted to again repeat I'm very impressed with Y-JS (and say that it is not getting the attention it deserves for this being on top of HN, given that it is what is novel, not the other stuff). But I also wanted to inquire more about the generalizable-ness of Y-JS and then hear about any specific specialized-CRDT you may be using for P2P rich text, because it looks like it could be a breakthrough! Would love to hear more! Excellent work.

Hi Mark,

Congratz on developing Gun! I've looked at it before and I was super impressed! The decision to use Y.js instead was done due to its modularity. We added IPFS support by simply publishing a connector https://github.com/ipfs-shipyard/y-ipfs-connector which Kevin Jahns later added to the list of connectors - https://github.com/y-js/yjs/issues/77

We have linked Y.js from the page header, if you click the word CRDT it takes you to Y.js directly. Nevertheless, I agree that we need to make it more obvious How we appreciate all the work that Kevin has put into Y.js :) Do not worry, PeerPad nor our CRDT usage is finished, there will be a lot more! (In fact, PeerPad was built with little less than 4 weeks of actual dev time, there is so much more we can do).

I'm following up by email, I'm interested in testing Gun as well. Meanwhile, join the CRDT discussion on the Research CRDT (https://github.com/ipfs/research-crdt) Repo, would love to have your input there.

Cheers!

It's good you welcome contributions to your open platform. In case your next demo is based on my work, please notify me in advance.

Hi Mark,

Thanks for your interest!

Off the CRDT libraries we analysed, Y.js was the one we picked up as it very modular, so that we could create our own connector and database layer.

These are all open-source: https://github.com/ipfs-shipyard/y-ipfs-connector and the encryption layer wrapping the database adaptor: https://github.com/pgte/y-indexeddb-encrypted/tree/encrypted

About the richtextdetails, the CRDT is based off of Quill.js deltas (https://github.com/quilljs/delta), composed by an array of such operations.

Anyway, this is all interim work, and we plan on making this independent of any specific library by implementing a generic CRDT on top of IPFS DAG and Pubsub APIs. If you're interested you can follow / chime in here: https://github.com/ipfs/research-CRDT/issues/11

Here's a writeable link to test this out:

https://peerpad.net/#/w/markdown/39fhMVeoTgEVj2SjNARrv6/4XTT...

Where are documents stored? Browser cache on each client?

Currently, yes, it's stored in the client. We have some plans for being able to select a pinning service for making sure the data persists: https://github.com/ipfs-shipyard/peerpad/issues/90

Haven't looked at Y.js but does this have undo support? Yu et Al. Describes a CRDT [1] with undo support for editing.

[1]: https://link.springer.com/chapter/10.1007/978-3-319-19129-4_...

I see these websocket connections in my network panel.

wss://sfo-3.bootstrap.libp2p.io/ wss://nyc-1.bootstrap.libp2p.io/ wss://nyc-2.bootstrap.libp2p.io/ wss://ams-1.bootstrap.libp2p.io/ wss://lon-1.bootstrap.libp2p.io/

I'm going to guess this is to establish the connections between peers. So this isn't really p2p because while there aren't any intermediaries for data transfer, there certainly are to establish the data connections.

If you want a p2p system because you care mostly about removing servers as a bottleneck, then this may not be an issue, but if you care about removing servers because you don't trust intermediaries, then this only gets you half way there.

> If you want a p2p system because you care mostly about removing servers as a bottleneck, then this may not be an issue, but if you care about removing servers because you don't trust intermediaries, then this only gets you half way there.

This is, actually, truly peer-to-peer. Those are connections to the bootstrap nodes, you can set your own bootstrap nodes if you want but you do need to be seeded with a few initial peers to find new peers. If you want to establish a browser-to-browser connection, you will need some server to help setup the WebRTC connection; this is the unfortunate reality of browsers. However, you can (theoretically^) connect to arbitrary non-browser IPFS nodes over a websocket connection (that's what's happening here) without some "blessed" node acting as the rendezvous.

^Unfortunately, there's a bit of a wrinkle:

1. Most non-browser IPFS nodes don't listen on websockets by default. The transport is still experimental and and has some bugs.

2. If you load IPFS from an https origin, browsers won't generally allow you to establish connections to non-https websocket endpoints. Unfortunately, even if you enable listening on the websocket transport on your IPFS node, you'll end up listening on an http (not https) websocket. This is because IPFS manages the encryption of the connection itself (IPFS addresses, or "peer IDs", are cryptographic hashes of public keys) and doesn't use (or play well with) the CA system. To work around this, the bootstrap nodes have nginx proxies out in front that to handle HTTPs connections. Most (all other?) nodes don't.

So the discovery is currently via Bootstrap Peers and a Signaling server that can help connect peers together. A P2P network needs some way of bootstrapping and having a couple of stable peers for doing that is the simplest and works in most scenarios.

But, it doesn't work in all scenarios. In the go-ipfs version of ipfs (or running js-ipfs) we can use mDNS for local discovery without any bootstraps, but unfortunately doesn't work in the browser (yet! FlyWeb would help https://github.com/ipfs/in-web-browsers/issues/45). Some sort of gossiping could work as well but we're always glad to hear other ideas of how we can make discovery work without having to hardcoded bootstrap nodes or signaling servers.

Unfortunately browser technology sucks, WebRTC still requires a server to bootstrap the connections. It is incorrect to criticize this as not being P2P just because of existing limitations which immediately are removed if you were to run the exact same app in Electron or something. The best thing we can do is plead with browser vendors to improve this.

These websockets node you see aren't servers. They're nodes like any other node, except they have a well-known domain name. They don't do any work that a browser js-ipfs node wouldn't be capable of.

Reply on news.ycombinator.com