Build your own Google Docs with GoInstant's new OT API

This is nice!

WebODF[1] developer here. Commenting because the project is doing this sort of thing and I have decent experience writing OT. I'll give a short explanation of how Operation Transformation works.

"Build your own Google Docs" is a slightly brave claim - "build your own Etherpad" would be more accurate.

This is because their operation set is very simple; for rich text documents the operations simply do not contain enough information to transform them against other ops. At the moment, their API includes 3 operations:

    * Insert  
    * Format  
    * Delete

What I don't see:

Something like MoveCursor, for a start. From the source code of the markdown editor, it appears that cursor position synchronization is handled outside of the OT realm, likely by some other signalling. This is bad because the cursor positions do not get transformed at all - you can see this if you try to collaboratively edit on the same line in the demo - your cursor jumps around unwieldily.

Why this happens:

Imagine two users A and B concurrently editing inside a paragraph (a | denotes a cursor, belonging to B):

  <p>abc|def</p>

Now suppose A inserts "123" at position 3 (after c), and B tries to move his/her cursor to the right by 1 step. Assume that A's operation reaches the server first. Then server's and A's state is:

  <p>abc123|def</p>

because B's cursor is pushed to the right by the preceding text.

Then B's operation "move cursor to position 4 (after the d) reaches the server. But A's operation has not reached B yet. The state on B becomes:

  <p>abcd|ef</p>

And the state on the server becomes:

  <p>abc1|23def</p>

Now B's op is relayed to A, and A's to B, and A's state becomes:

  <p>abc1|23def</p>,

and B's state becomes:

  <p>abc123d|</p>.

because there is no OT for "MoveCursor" against "InsertText" -- in fact "MoveCursor" is not even an op here. This is an inconsistent state, which is basically a broken editing session (should never happen). Fixing this up by passing around diffs is not allowed (in a strict sense); diffing is not controllable OT.

WebODF uses stateless client-side OT, so any server is just mostly a dumb operation relayer. See what the OT logic for these two operations does: https://github.com/kogmbh/WebODF/blob/master/webodf/lib/ops/...

If the "InsertText" operation's position was lesser than the concurrent "MoveCursor" position, "MoveCursor" is transformed to increment it's position by the "InsertText" operation's text length. This ensures that the cursor, while moving, will end up at the intended position within the word you were looking at. (you'll notice that "MoveCursor" has a 'length' parameter in WebODF because a cursor also represents a user selection, yes it has colored collaborative selections).

OT'd process:

  <p>abc<cursorB/>def</p>

A inserts "123", A's state is:

  <p>abc123<cursorB/>def</p>

B moves cursor to position 4, B's state is:

  <p>abcd<cursorB/>def</p>

A gets B's op, transforms it to increment position by "123".length = 3, state becomes:

  <p>abc123d<cursorB/>ef</p>

B gets A's op, applies it because moving a cursor doesn't affect text insertion, state becomes:

  <p>abc123d<cursorB/>ef</p>

Hooray, Eventual Consistency!

Any OT implementation is full of such opinionated enforcements and tradeoffs. It should make some assumptions about the editing UX and make sure that this is preserved when doing realtime concurrent editing; ignoring that and only going for eventual consistency will result in situations where the user cannot look at where they are typing.

WebODF has many such operations (~23) for each kind of edit: "SplitParagraph", "InsertText", "ApplyHyperlink", "ApplyDirectStyling", etc at last count. A large number of them are supported in collaborative mode, i.e. they have OT written. And this is no small task, because each operation needs to have a transform written against every other operation in the set of all operations, including itself. It does help that not all operations interfere with others, so the OT matrix is somewhat sparse. :-)

Fact: Any complex rich text editing framework will need to have it's own customized implementation of OT. We're dealing with the Open Document Format, we have Operations that are significantly tailored to it.

I feel that GoInstant's API is a nice thing! But it needs: 1. More operations. 2. Purely client-side OT (not strictly required, but very very useful).

Shameless plug: WebODF is a FOSS project with ~3 fulltime developers (including me) working on it. We are also working with the ODF change-tracking subcommittee to standardise things. We need more contributors and/or donations. :)

  [1] http://webodf.org/

EDIT 1: Sorry, this post became longer than I had wished. I hope it helps anyone understand how OT works. Also, 'gratulations to GoInstant for launching this. :)

EDIT 2: Reformatting post for readability.

An Operational Transform API has always been my "maybe when I have more time and knowledge" idea. I'm glad to see someone else doing it and I hope I have a chance to use it. OT is the magic behind Google Docs, and it's one of those really elegant protocols that most CS people instantly appreciate when it's explained to them.

http://en.wikipedia.org/wiki/Operational_transformation

I can recommend this article that explains OT really well: http://www.codecommit.com/blog/java/understanding-and-applyi...

Google Drive Realtime API developer here. https://developers.google.com/drive/realtime/

If you're trying to build your own Google Docs, you might want to consider using the Drive Realtime API, which is based on the same technology as Google Docs.

The Drive Realtime API has a wide range of mutations built in for common data structures: - Maps (last writer wins) - Lists (insert, delete, overwrite) - Strings (insert, delete, overwrite)

We also support arbitrary data structures including trees and graphs.

For modeling cursors, we have "Index References" which are effectively pointers to a specific element in a string or list. You can move them around on the client side and we have all of the transformation logic worked out so that concurrent edits don't make them point at the wrong position.

Because the data is stored in Google Drive, you can create, share and organize files using the standard functions in the Google Drive API. There's also an easy-to-use REST API for importing and exporting data as JSON.

Are there any great open-source implementations of OT? Last time I looked there were some, but the quality and popularity of them was questionable.