Realm Mobile Platform – Realtime Sync Plus Fully Open Source Database | Hacker News Reader

Realm Mobile Platform – Realtime Sync Plus Fully Open Source Database

realm.io

222 pointsmlschmitt2375 comments

Switched from CoreData to Realm for a database-heavy communications iOS app. Been using it for about 6 months, so not an expert, but I've done some interesting stuff with it :)

Main benefits I saw:

  * faster performance
  * simpler multi-threading
  * built-in encryption based on ios' commoncrypto (fips 140-2 module)
  * fast response time to questions on github (in 3 cases)

Main drawbacks:

  * no icloud syncing. I need to check out the new syncing stuff...
  * new-ish database so does not have ALL features (e.g., multi-process encrypted access). They appear to work quickly to implement features the community cares about.
  * performance is fairly "magical". I understand normal database query performance, realm is different...
  * No in-place VACCUUM-type functionality. I've had issues with the realm database file growing unexpectedly large.

Hi! Tim from Realm here. Glad to hear that Realm's been working out well for you in place of Core Data. :)

In response to your drawbacks there: * Definitely try out the Realm Mobile Platform. We're also looking at leveraging CloudKit to seamlessly authenticate all devices belonging to a user. * Multi-process encryption wasn't possible until relatively recently (Issue #1845 on our newly open-sourced realm-core project!), so we'll definitely be looking at it from now. Please do thumbs-up any of our GitHub issues for features that you would like. * We've published articles on how the Realm Core works in the past, but now that it's open-source, you can go and see the code for yourself! * It's possible to generate compacted copies of Realm files, so you can implement this functionality yourself if you need it. We've had long discussions about whether this should be an automatic feature, but we feel in most cases it would be premature optimization.

Former Realm employee here.

So glad to see the team finally launch this. I've seen them work their asses for years (literally!) to launch something I know Realm’s 100k+ dev community was clamoring for from day one but was really hard to build.

I'm biased of course but honestly I think they have soft-key shipped one of the most transformational mobile technologies ever: real-time, conflict-free sync that works as well offline as online. There was a lot of literature on this of course but bringing it all together in one product was something else!

From engineering to marketing, product & founders I know they've sweated to get here and I'm just so, so happy for them.

I know they will make a lot of developers and customers happy, and for that I say: bravo.

CouchDB and iCloud already have this.

iCloud's implementation is awful. It doesn't help you understand or resolve the conflicts at all and just assumes last write wins.

Is such a conflict likely to occur with a single user?

@timanglade, congrats to you and your former team! Always awesome to launch, good job.

Full disclosure, I work on a competing product (MIT licensed, https://github.com/amark/gun ) and do a lot of testing on distributed systems.

You guys say it is offline-first with conflict-resolution. However your documentation ( https://realm.io/docs/realm-object-server/#conflict-resoluti... ) states the "Last update wins". This is contradictory for two reasons:

A) Either you mean last write wins relative to the server. But an offline update is going to get sent late (on reconnect), which would make it "last" and lead to weird problems.

B) Or you mean last write wins relative to the time stamp of the client. But this does not account for wall clock drift. What happens if the client's clock is intentionally set 2 years in the future? In a distributed system, everybody has a different "last".

The demo video, while very well done, uses a drawing app which does not really demonstrate real conflict (ie, if 2 users draw over the same space, it becomes a z-index rendering issue, which could be solved with conflict resolution but isn't a good example of conflict).

See the full answer from Realm’s founder/CEO here: https://news.ycombinator.com/item?id=12590753

Thanks, did not see that. Not all vector clocks provide conflict resolution though - actually, they can very much lack in this regard. For instance, you are much more likely to get two clients that increment to the same vector while offline than two clients getting the exact same millisecond timestamp.

Which would make the problem worse, not resolved. What type of vector are you using?

Simon from Realm here - just wanted to chime in and clarify some details on this. It would be more correct to say that we are using Lamport timestamps, with the addition of a globally unique ID for each client and some tracking of changeset ancestry. No two identical [timestamp, id] tuples will ever be produced in our system, so there is always an unambiguous way to decide which side "came first" in case of a causally unrelated conflict.

It's important to understand that most "conflicts" in our system are resolved without resorting to timestamps at all, which is possible due to the way we encode our changesets on the wire and the fact that we track their ancestry. It is only used in cases where we absolutely have to, i.e. when there is no causal relationship between two updates of the same property (in which case, the "later" timestamp wins).

Hope that answered your question. :-)

Awesome. Thanks!

Thanks for the details, just wondering is the implementation mentioned here available to review somewhere? Is it included in the realm core repo?

Is this documented somewhere? Understanding how sync works here and under what circumstances the timestamp is required to resolve conflicts is critical to understanding how to design correct applications based on this.

How do you deal with security/access control?

This seems to me the most difficult aspect of a client-side database, but please correct me if I'm wrong.

Realm supports AES-256 encryption so you can encrypt your data at rest and then SSL/TLS for security in transit.

Realm Object Server supports the ability to control which users can access which Realms, supporting for example shared Realms among users to power collaborative experiences like the drawing app in our demo video. The client side APIs to manage these permissions weren't exposed in this release but will be coming soon in the beta.

If Firebase releases an on-prem or self-hosted server, what does the Realm Mobile Platform do differently?

I'm really excited about Realm, but I honestly can't read this without asking (rhetorically), "have you ever heard about Firebase?" From a technological perspective I think it's unfair to claim Realm is the first to the party here, Firebase engineers have shipped an equally impressive technical feat. I understand the vendor and platform lock-in with Firebase is a _major_ differentiator when compared to Realm - but the underlying technology appears to offer the same benefits.

(Adam from Realm) First, we respect everything Firebase and team has done. Others might feel differently, but shipping great software is hard, so nothing but respect from us there.

Second, and to address your question, we are very different architecturally than Firebase. We built from the ground up an object database that runs on mobile. As a result, this means when using Realm Object Server, the mobile development is still focused around that database. In practice, this means developers only have to think about local APIs because their UI is now bound to the data in their local Realm, with the networking handled automatically. This isn't just a basic cache, though, but a full-featured database that includes: memory-mapped object interface, ACID compliance, querying, and much more. Firebase doesn't offer that capability since it's mobile SDKs aren't built off a mobile database like Realm.

Beyond the mobile capability/developer experience, the server-side capabilities of Realm Object Server go beyond whether it is self hosted or not. While not included in the developer edition, we offer server-side SDK support and event handling capabilities so that the server can integrate into existing infrastructure vs attempt to control how you build your backend.

Or, you know, support an open source alternative like PouchDB/CouchDB. Sync, conflict resolution, offline-first, and data push are all supported. Not to mention it's a very stable, mature ecosystem.

There are even enterprise-ready setups using CouchDB (like Cloudant) for those that want it.

Hi, former Apache CouchDB guy, former Cloudant employee, and former Realm employee here… I (still) love CouchDB but at the end of the day I think its sync is a better fit for server-to-server or web client-to-server use-cases, not mobile-to-server.

There’s tons of stuff CouchDB sync doesn’t support that end up being a huge problem with mobile apps… Here are the top 3 things Realm adds in my opinion: great client-side performance, native models that are extremely easy to use, and conflict-free sync (not conflict resolution!). With my own eyes I’ve seen a generation of developers try CouchDB for mobile apps and then abandon it because of the limitation of its approach. Personally I’m glad to see someone try to bring more options to developers. I think this is the first general-case true sync (à la Google Docs) for mobile, and I hope it will continue to spur more iterations & innovation in other open-source projects as well.

Thanks for listing those potential benefits. Any idea if those CouchDB issues are covered in detail somewhere?

I work on PouchDB, in terms of those points:

"great client-side performance": PouchDB is more than fast enough for most use cases, its just a wrapper over indexedDB. (there are areas we can improve, particularly performance of map reduce / mango queries).

"native models that are extremely easy to use": Not sure what that means, PouchDB and CouchDB use json, its native and easy to use in JavaScript

"conflict-free sync (not conflict resolution!)": Certainly going to have to take a look at this, any conflict free sync protocol I have seen put a lot of limitations on how data within your application is structured but conflict resolution certainly is something that can be improved within Pouch / Couch.

I do think there is a bunch we can (and are) improving in PouchDB and its great to see alternatives, will be looking into this further now. But I certainly do not agree that (C|P)ouchDB are not suited for mobile <-> web sync, thats the primary use case I work on Pouch for.

"native models that are extremely easy to use"

This means native code (specifically not JSON). So in Java you would be dealing with POJOs, etc...

A bit OT, but I have to compliment the person who wrote the docs[1] for PouchDB. They're quite entertaining and easy to understand. Well done!

[1]: https://pouchdb.com/guides/

Thanks, I'm that person. :) I tried to write it as a "missing manual" for Couch/Pouch, and had a lot of fun in the process! It needs some updates for newer stuff, though: pouchdb-find, custom builds, Chrome DevTools storage UI improvements, etc.

(I also work on PouchDB.)

For "conflict-free sync," according to the docs, this appears to be last-write-wins along with a special case for deletes: https://realm.io/docs/realm-object-server/#conflict-resoluti... . (Incidentally Couch/Pouch also has a special case for deletes, but it does the opposite behavior by default – deletions lose to updates.)

There's also some language in here about how "insertions in lists are ordered by time;" I'd be curious to know how they coordinate clock times across nodes. Ditto for "last update wins" – important to know how "last" is determined.

In general, I tend to be skeptical of systems that hinge on last-write-wins, but LWW is definitely a model that works in a lot of cases where you're not too concerned about the possibility of losing data, and it's also a simpler mental model for the app developer.

Realms approach to conflict resolution is very different from what you see in products like Couch. Realm is an _object_ database, so in contrast to document databases where conflict resolution happens very coarse grained at the document level, in Realm it happens at the object level (and actually all the way down into individual properties).

This essentially means that the entire object graph is treated as one big CRDT [0], transparently handling conflict across individual objects, properties and relations.

When talking about ordering by time, both in the context of LWW and insertions in lists, we are talking about vector clock time, not necessarily device time (or though that is taken into account as well).

[0] https://en.wikipedia.org/wiki/Conflict-free_replicated_data_...

LWW is a tradeoff. You'll be better off selling it as an automated-resolution option on top of manual resolution, instead of an innovation over manual resolution. (MV registers are a CRDT as well!)

An-object-graph-as-a-CRDT is a really interesting concept. So far, I am not aware of any literature formalizing/studying it. I can only recall the recent CRDT JSON paper by Kleppmann et al, which is significantly less ambitious. Is your approach documented?

While reading the announce, my first guess was some recursive per-field LWW, but you mentioned vector clocks. That is intriguing.

Can you explain this statement: "There’s tons of stuff CouchDB sync doesn’t support that end up being a huge problem with mobile apps"

You lay it out there and then talk about what Realm adds, but never what Couch lacks. And are you including PouchDB in your evaluation of CouchDB for "Client-Side Performance"?

As I understand PouchDB is performant, uses JSON (pretty much the definition of "Native Models"), and allows server-arbitrated conflict resolution with stored revisions, making the simplest conflict resolution "last-in-wins" with user UI for "undo" (or any JS-object merge library).

Thanks!

I think you're misunderstanding what "Native Models" means in the context of iOS work. Saying that a database uses JSON immediately, to many ios devs, recalls years of writing very fragile json serialization code. Looking through CouchDB's iOS example apps confirms that yes, basically everything that isn't a string or int has to be serialized into JSON. That's a pretty enormous annoyance, and incorporates a lot of additional cognitive overhead when dealing with data that Realm simply handles properly for iOS from the beginning.

I'm deeply relieved to see that they're launching something that does something about the fragile ios architectures built around JSON. It's similar to what they did with their mobile database: they actually listened to what mobile devs found annoying, and then they built something that completely sidestepped all the fragility problems that emerge when you wrap sqlite with an ORM. Instead, they made an approach that fit within our workflow, as opposed to taking us further from it.

Ahhh, got it. When I hear "offline-first" my mind's been in webdev-mobile, not appstore-mobile. Thanks for clarifying!

CouchDB doesn't have any iOS code, are you perhaps referring to Couchbase?

Ohhh my god, you're totally right. I was looking at this: https://github.com/couchbase/couchbase-lite-ios

The things Realm adds are what CouchDB lacks. Those things are essentials from a certain point of view.

How is "conflict-free sync" done? The usual approach is last-write-wins, which is just a euphemism for "randomly losing user data", or CRDTs or Operational Transforms, which severely limit the data models one can sync.

I'm happy to learn about a silver bullet here.

CRDT, but also vector clocks.

https://news.ycombinator.com/item?id=12590753

Some CRDTs employ vector clocks. CRDT is a veeeeery broad family of algorithms and data structures.

Is that not what I stated (even if it was poorly phrased, and I can no longer edit it)?

It is true that most Operational Transform systems have limited data model support, but that's an area Realm is quite unique on. The Realm Mobile Platform supports all of Realm database's current data types, such as native support for links and ordered lists. Also have plans for much more including: substring changes, counters and sets.

Thanks, that's gonna be the limiting factor then, developers not finding a way to represent some of their storage needs on this platform.

It's a fair trade-off, but hardly neither objectively worse or better than couch-sync. I'm excited to see where Realm goes.

In my very unscientific approximation couch/git-style sync solves about 80% of general use cases, CRDT/OT cover another 10-15% and the rest are poor souls that have to do something custom. (All this excluding binary sync/rsync/BitTorrent/Dropbox.)

Oh, and while there is still room for improvement, we've improved the server-mobile sync case by an order of big-O-magnitude. And this is a first class design target for the future.

Adam from Realm, we appreciate being in good company! Our approach differs in building off our object database. We couple this with low-latency sync that only transmits changes and deterministic conflict resolution, making collaborative experiences really easy to build. Still lots to learn while we are in beta, but we also announced open-sourcing our database core today as well!

Congrats on the big release.

I remember I was using very early version of Realm in the one of the Android apps I was working on at that time. It was not working properly for me, unfortunately. Random Segmentation faults and limitations that one have to use the POJO extended from the RealmObject. So, I eventually switched back to SQLite (plus ORMLite).

I have seen a lot of changes and now I think I would definitely consider giving it a try.

One question that I have is even though it's nice to have self-hosted Realm Object Server, are there any plans or timeline that it's going to be open-sourced ?

Paul from Realm here: The Developer Edition of Realm Object Server is free forever, and we're committed to adding new features and making it more powerful over time, but at the moment we haven't made any decisions about making it open source. We do have a strong history of open sourcing the things we build, so it's definitely something we're thinking about.

I found Realm to be very easy to use with React Native. I had trouble using the other tools to save data inside my React Native apps. Anyone else been in the same situation?

I'm using it in my React Native app. Pretty happy with it so far. Haven't tried anything else, as Realm looked like the best choice at when I was selecting the data store for the app.

Any updates on the C++ API and lib? It's been pending since a year[1] and there are no updates :-(

[1] https://github.com/realm/realm-cocoa/issues/2198

We open-sourced our core which is written in C++, though different API. This now makes it easier and we would love to build it alongside the community: https://github.com/realm/realm-core

Kudos!

This is great news for us Mobile C++ developers who don't have alternative to SQLite.

You just made my day :-)

Definitely going to try this out. However, I don't feel great about not having...

- any concept of clustering for reliability and failover - any easy/clear way to distribute load to the object server - an easy way to run backups

I may just be blind in the docs. I really don't want to go back to running my own database server. Services like compose.io are soooo nice. Clustering, automatic failover, backups...

Another Core Data alternative to try is YapDatabase: https://github.com/yapstudios/YapDatabase

It is similar to Realm in a lot of ways, with a simple concurrency model, and native objects (key/value/collection). It's built on top of SQLite so you can use extensions like secondary indexes, full text search, RTree. You can also use SQLCipher for full database encryption, and there's another extension to do automatic syncing via CloudKit.

Caveats are it only runs on Apple platforms, doesn't have cross platform sync, and isn't backed by VC funding. If you need those things then Realm is definitely your best choice.

Bump for Yap. I've successfully shipped an app that used this with the CloudKit sync extension.

Have to say the mappings / view paradigm was not very intuitive starting out, but once that clicked things worked very well.

Glad to hear Realm is providing an alternative for syncing and will have to try it out.

So this is like alternative to Firebase? How different is their offering?

Christian from Realm here. Main differentiators are probably:

- Realm is self-hosted compared to Firebase which is an MBass.

- Native objects all the way down vs. a JSON structure

Congrats!

Realm and React Native is now my go-to, keep up the solid work :)

Congrats! This is a major step forward which we will be supporting in our product. Its right now being used by close to 500k users :)

Question - TLDR - Is p2p synchronization possible?

Two databases in two different devices in the same network, is it possible to achieve synchronization without having access to a server?

I wanna build a POS app and need a solid sync experience. I'm building my own but obviously prefer to use something else ;)

This could support multiple users against a master database editing and getting up-to-date(as possible) data? And sync reliable?

Sure looks like it does.

Yes it would be possible to support. This version doesn't expose adjusting Realm permissions such that multiple users can have read/write permissions to the same Realm. This will be coming during the beta and exposed as client APIs for users to manage permissions to Realms under their control.

It says the database is open source but I can't find any link to the source code. Is both the client and server open source?

The core database that powers all the versions of Realm was open-sourced today here:

https://github.com/realm/realm-core

The various SDKs that use the core for each mobile platform have always been open source, such as realm-cocoa:

https://github.com/realm/realm-cocoa

> The core database that powers all the versions of Realm was open-sourced today here:

Ok, what about the server database?

The server runs the exact same database. It is only the sync part that is proprietary.

> It is only the sync part that is proprietary.

Any reason for that? It's also doesn't look very straightforward how exactly the database is deployed and accessed on the server. As it stands now only the 'client' is open sourced but if someone wants to hosts its own `realm` database it still has a lot of work to do. I really find this misleading when various SAAS vendors claim they are open source when in fact they just provide you just a client to access their service. Nobody(almost) would use a `binary` client anyway.

Can someone explain why use Realm, or CouchDB, over SQLite?

Speaking from Realm's side, the primary benefits most developers give are:

- Object-based: not an ORM

- Speed: Realm is quite fast and in some cases faster than SQLite

- Ease-of-use: a lot less boilerplate code and a simple threading model make it easy to get going with.

Any plans for a browser version?

It is something we want to do but right now we are just focused on mobile. Would love to learn more about what you might like in a web framework: af@realm.io

We've updated the link from https://realm.io/products/realm-mobile-platform/ to this introductory page.

These guys seem to be copying the Parse business model. Which as we all know worked out very nicely for the founders and investors. But very badly for anyone that adopted the Parse framework.

Very legitimate concern. I use the Realm database in production and it has proven to be a very good decision. Huge advantage over Parse is that the database is open source. If something did happen, the open source community would probably be enough to carry it. If not, it is just a local persistence store, so not hugely difficult to swap out for something else.

This new mobile platform is not (as far as I can tell) open source and has completely unknown pricing beyond the free tier. I certainly wouldn't go near it, even though it looks great.

Well, I'm not sure about the pricing, but according to this page (https://realm.io/products/realm-mobile-platform/) you can host the mobile platform on-premises.

Yes and no… I mean it seems like Realm is more focused on an “offline-first” approach by contrast, and is adding this "database-as-a-service platform” for platform agnostic sync.

But, yeah… the usual “what happens if I run into a problem they can’t solve for me [in a timely manner]” or “what if I want to make hard-optimizations” or “what if they break something for me” or “what if they sunset it and I’m building an actual business on top of it” questions apply, of course.

In general, depending on what you’re doing you'd want to MINIMIZE your dependencies on services like this. Luckily, they’ve always seemed focused on database more so than “everything” like Parse was… so it’s only 1 thing, albeit perhaps the most critical one.

Alexander from Realm here.

We are very aware of how users of Parse got left in the cold when Facebook dropped the project. That is one of the reasons that we choose to offer Realm, not as a service that can be shut down, but rather as a free download that you can self-host on any provider.

That way, even if something happened to us you would still be able to continue running your apps on your own servers. I know that there could still be concerns about how to maintain the software, which is why we have also open sourced the underlying database.

I think most people will agree that we have a pretty good track record of being part of the ecosystem and being there for the developer community, and we very much intend to stay that way.

I disagree that Parse worked out badly. Now that parse-server is open source, the ecosystem is stronger than ever.

I hesitatingly and begrudgingly used parse while it was still hosted by Facebook. At the time my biggest and only problem with it was the vendor lock-in. Now that it's open source, that problem no longer exists, and all the great tooling is still around.

I'm happily using it on new projects. I suggest taking a look and trying it yourself.

Reply on news.ycombinator.com