I'll need to think about this.
Maybe do the routing through a distributed hash table like we use for finding torrent peers (perhaps even the exact "mainline" DHT torrents use), which would mean that even with the server being down you could still sync with your other online clients.
The main problem is garbage collection - to guarantee that you can sync across all devices for all time, your data structure must be append-only, so your document can only grow. A long living document will eventually grow very very large. You could let the user decide when to collect garbage/establish a new baseline doc; maybe like 5 years. But this means that if you edited the document on a laptop, didn't sync it with another client, and then closed the laptop for 5 years you would lose the ability for the changes you made on that laptop to be resolved automatically.
But not all! Quite a few that are state-based, auto-compacting, space-saving, small & efficient!
More info in a discussion on this: https://twitter.com/marknadal/status/1008610024875122688
But you're right that I'm wrong when I say "[all] CRDTs are generally append only structures." I meant to say "text-revision CRDTs are generally append only data structures."