Show HN: Realm (YC S11), a mobile database | Hacker News Reader

Show HN: Realm (YC S11), a mobile database

realm.io

188 pointsbjchrist96 comments

The foursquare map video shows how the experience with realm is better (and does that well), but it didn't give me any sense of how it's achieving those results. As a developer, I'd love to see an accompanying explanation with details on how the original version works, and how the realm version works.

Absolutely. We didn’t get a chance to clean up the codebase for this app before the launch (it’s actually using an old version of our API that would be confusingly different than what our docs detail) but we’ll try to set up a blogpost about that since I may not be able to do the topic justice in a comment here :)

The short version is that most apps tend to do a query on click, fetch from an API and then cache the results in memory. The Realm version fetches from the API asynchronously in the back, pre-fetching results and writing them form the DB on one thread and reading & displaying them on the map from another. This has a few benefits (& drawbacks). On one hand, you do hit your servers a bit more often at first, and you may actually get so much data in the map that the UI component will start to lag. On the plus side, your data points are cached so you can do a lot less API queries over time (especially for something like Foursquare where users tend to stay in the same area and the dataset of venues doesn’t change that often). You also get better control as a designer or developer, and you can trim down & display relevant datasets on the fly.

To sum up, I’d say the key difference is the availability of a local, concurrent data structure that can easily be updated & read from multiple places (i.e. in a few lines of code if you look at the samples on http://realm.io)

Thank you! So in theory one could do the same thing writing your own threads and using vanilla local storage? But realm is removing all the boilerplate custom code you'd have to write yourself, and making the local side more performant than custom code would be? Is that it in a nutshell?

Yes, that’s the spirit, although I’m sure others would tell you that using Core Data across threads and at high throughput it much easier said than done :) You can achieve anything with almost any storage technology — we just try to let you do it well, easily and in a very performant & maintainable way.

"You can achieve anything with almost any storage technology" - is actually being very generous to quite a few other technologies :-) One of the promising benefits of any technology that can make things run magnitudes faster, is actually that you can do totally new things you previous thought impossible.

I'm looking for exactly this. Thanks for building.

LOVE this. I've dealt with the pain of multiple CoreData contexts in different threads and ugggh. This was easily the longest part of our Streak app development, as we needed it to be fast, offline, and fully synced. This is definitely a way better API that accomplishes the same thing as CoreData. If its actually faster too, wow, just amazing.

One question - CoreData works really well with built in UIViewControllers like TableViewControllers. Any chance you'll be releasing some code that makes wiring up changes in the models to the controls that are displaying that data?

Thanks for the kind words! Regarding wiring this to a TableViewController, we have some code samples already for Swift[0] and Objective-C[1]. Right now you get notified of changes and refresh, but we’re very interested in exposing a reactive-style binding mechanism as well. The storage engine underneath supports that very well actually, so stay tuned…

  [0] https://github.com/realm/realm-cocoa/tree/master/examples/swift/RealmSwiftTableViewExample
  [1] https://github.com/realm/realm-cocoa/tree/master/examples/objc/RealmTableViewExample

The main benefit of using CoreData is seamless faulting behavior. It's not obvious from the example or the site that Realm supports something comparable. How does Realm deal with a lot of objects?

Realm does not have a concept of faulting like Core Data does. In Core Data every object is essentially a copy of a representation in the underlying data store (usually sqlite), so instantiating the object is quite expensive as all the properties have to be retrieved and copied into the object, and before you have done that it is very limited what you can do with the data.

In Realm the object you see is a direct representation of the underlying data, it does not have to copy in all the properties. When you access them, it is the property directly from the db you are getting, not a copy stored in the object. This means that it takes up a fraction of the memory space, and it also means that queries and similar operations can work on the objects directly in the db, without having to instantiate them in objective-c. This means that you can easily scan over millions of objects, without causing a single "fault".

When manually traversing objects you still have to instantiate them, but since they don't have to copy any data, it is a far more lightweight process than faulting.

What about support for Windows Store and Windows Phone? Those platforms need something like this too.

Sorry one more question - Android?

Android is coming! We have an internal build that’s almost at parity with iOS but still needs a bit of tuning and is missing a few key features like Migrations. We’ll work to release & open-source it soon. You can register to be notified of that [0].

[0] http://realm.us5.list-manage1.com/subscribe?u=2aab5198c2f56b...

Awesome.

As good as Realm looks for iOS, I think it might be needed more in Android as there are no really viable alternatives other than using sql lite directly.

Android is in the works! You can signup to be notified of progress : http://realm.us5.list-manage1.com/subscribe?u=2aab5198c2f56b...!

Wow - this looks really great! As someone who has struggled with Core Data in the past, the API looks great and things seem, on the whole, a lot cleaner. Looking forward to giving it a go.

EDIT: I just came across the REST API / JSON import section in the documentation - really cool stuff!!

The API approach vs. SQLite, etc. is interesting - particularly that it does not look to have very many moving parts. I'm curious about transactional semantics though. Are they 'always on' or is there API control (still haven't gone through docs completely yet, so I may answer my own question later).

Alexander from Realm here, all interactions with the database is through full ACID transactions, but it is a bit untraditional in that the interaction model is build specifically to mobile apps.

The transactions interact with the runloop, so that you are always in an up-to-date read transaction (which always give you a consistent view and never blocks, even when other threads modify the same data). To write, you have to do explicit write transactions.

So your data is always live, consistent and up-to-date without you having to do any explicit coordination (apart from the write transaction on changes) and notifications allow you to always keep your UI up-to-date with the latest changes.

For people using SQLite, it's trivial to do the same with WAL mode and a read-only transaction on the main thread (which is what you should be doing). All writes happen off the main thread, so performance is great.

I could do with some clarification on what problem the product solves, and actually, what the product even is. To me this looks like a new DB, and an ORM style interface to interact with it?

I like that it is multi-platform, but SQLLite is a pretty mature DB store available on all the major platforms anyway. I get that it is faster... but presumably there are there trade-offs for this speed gain. I would like a better understanding of what they are. Transaction support?

(PS: I would also much prefer if your objects can be typed in a way so they are plain, rather than forced inheritance from RLMObject)

EDIT: The blog post at http://realm.io/news/introducing-realm/ answers some of these questions.

Absolutely. There’s a lot more details in our launch blogpost [0], but you got the right impression: it’s a custom C++ storage engine underneath (not SQLite), that is tightly integrated with the object layer. Not an ORM per se, because the engine isn’t relational, and because there’s not copying/mapping/translation going on — the data on disk is the same data you manipulate in your language.

The big problems we try to solve are really ease of use & performance. We heard from many developers dissatisfied with the current options on mobile, and the relative complexity to accomplish tasks like accessing data across threads, JSON handling, performance on graph queries, etc. — which the reactions on Twitter seem to confirm. We’ll write a lot more about the tradeoffs in coming weeks, but we’ve really grabbed a lot of perf cycles just by taking advantage of modern techniques such as zero-copy architectures, bitpacking and vectorization. Of course the big implicit tradeoff is that you give up a bit of the relational paradigm in exchange for that speed, although we do maintain ACID transactions, immediate disk persistence by default and schemas.

Completely hear you on wanting plain objects — I personally feel that way too! Unfortunately, there’s no way to achieve what we do without making you inherit from RLMObject, although we do try to make those behave as close as possible to NSObject.

[0] http://realm.io/news/introducing-realm/

Although I don't know exactly what the implications are just yet, I've created a category on one of my RLMObject subclasses which accepts an NSArray or NSDictionary property and in the getter/setter converts to and from the NSData backing property. The only thing is after [Venue creatInDefaultRealmWithObject:] I have to explicity set the value for it to propogate to the backing NSData property.

It's extra work, but it lets me manipulate Realm into doing what I want it to do to store data which doesn't conform to an explicit schema. (e.g. a Venue which has an array of address strings, where I don't want the headache of iterating through venue.address[i].string values:

(note: this doesn't look pretty, its doing some funky formatting!)

"venue" : { "name":"Test Venue", "address" : [ "address line 1", "address line 2", "address line 3 for rare instances" ] }

@interface Venue : RLMObject @property NSString name; @property NSData addressData; @end

@interface Venue (AddressToData) @property NSArray address; @end

+ (NSDictionary )defaultPropertyValues { return @{ @"addressData":[NSJSONSerialization dataWithJSONObject:@[] options:0 error:nil] }; }

- (NSArray) address { NSArray address = nil;

    if (self.addressData) {
        address = [NSJSONSerialization JSONObjectWithData:self.addressData options:0 error:nil];
    } else {
        address = @[];
    }
        
    return address;

}

- (void) setAddress:(NSArray )address { if (!address) { address = @[]; }

    NSData *data = [NSJSONSerialization dataWithJSONObject:address options:0 error:nil];
    
    self.addressData = data;
}

RLMRealm *realm = [RLMRealm defaultRealm]; [realm beginWriteTransaction]; for (NSDictionary *v in json[@"venues"]) { Venue *venue = [Venue createInDefaultRealmWithObject:v]; NSArray *address = v[@"address"]; venue.address = address; } [realm commitWriteTransaction];

In my experience object persistence rarely needs SQL or other "relational additional stuff" underneath. Typically I need to store objects and their mutual relations persisted as if they were just kept in memory. Have often thought about how weird it was that you could not just persist objects "as they are" without additional mappings etc. I saw a demo of Realm and had a talk with one the developers a couple of weeks ago. As the demo and our talk progressed I found myself nodding more and more excited. As I understand Realm it seems to be based on a very natural and straightforward principle. It can't NOT be efficient... I can't wait to try this out in a real project!

I was going to say Android already has SQLite built-in, but they do a good job explaining advantages vs that (more object like syntax, more document like vs. table and schema like, better performance, etc.). Good job.

What does this provide that CoreData doesn't? Whatever it is, it needs to be clearly explained on the landing page. Otherwise it's difficult to understand what problem this solves compared to CoreData.

The big thing is really ease of use. Our blog post has a few more details on that point [0] but basically what we heard over and over and saw ourselves, is that Core Data quickly gets in your way if your app grows [1][2][3]. In essence, we tried with this initial release to offer a rich feature set with a simple API, less side-effects (especially for multi-threaded apps), and a performance that’s even better than using raw SQLite.
In early testing we got a lot of higher-level questions about what Realm was exactly, so we decided to go for a descriptive tagline instead, but you’re right that we should add more details about how we compare to Core Data!
[0]http://realm.io/news/introducing-realm/ [1]http://wbyoung.tumblr.com/post/27851725562/core-data-growing-pains [2]http://alastairs-place.net/blog/2013/04/17/why-core-data-is-a-bad-idea/ [3]http://inessential.com/2014/06/11/core_data_thoughts_june_2014
(EDITS: formatting)

Core Data has been improved and stabilized by Apple for over a decade, is used internally by many first party apps in iOS, is well documented and fully integrated into Xcode, and heavily "marketed" by Apple at WWDC. Any perceived performance issues can be eliminated or reduced by using faults, indexes and improving queries through NSPredicates. Even one of the blog posts you quote above [3] states that if he was starting a new app today, he'd go with Core Data. So I'm just not seeing a need for yet another ORM solution. I'm not saying you shouldn't continue building it, but you guys need to do a better job explaining the differentiation against Core Data. One area could be sync, where Core Data still needs work, and is limited to iCloud's backend. Perhaps that could be your differentiator? But you're not talking about sync yet. So there's work to do in crafting your message. Fwiw, I'm a senior iOS developer who evaluates these things for a living.

Just to be clear on the topic, since this is an easily misunderstood aspect of Realm: we are not an ORM. We’re not built on SQLite, we’re not mapping to a relational layer, nor are we serializing objects from a third-party data representation: your objects in the language are exactly the same data that’s stored on disk.
We heard a lot of dissatisfaction from the community as far as Core Data is concerned (including from Apple engineers themselves). I’m also not entirely sure you could ever get Core Data performance to the levels we enjoy, since Core Data cannot be any faster than the SQLite underneath, after all. I do want to say that your feedback on how we present ourselves and the work ahead of us is spot on and much appreciated. Many thanks for speaking up, we will work hard to heed your comments & improve.

Just as another data point, I agree with magsafe. I've used Core Data a bit recently and haven't really had any problems with it (though I agree it could be nicer).
My huge, huge, huge problem, on the other hand, is sync. I don't want to get locked into iCloud and the idea of Apple offering multi-platform support is unrealistic.
If you offered this as a cross-platform solution + syncing I'd sign up immediately (it sounds like this might be where you're headed anyway).

Great attitude, and thanks for the clarification. Best of luck!

Anyone using CouchbaseLite?
http://developer.couchbase.com/mobile/#couchbase-lite
I see a lot of comments here about wanting a CoreData replacement that's better at sync, which is exactly the use case CouchbaseLite is meant to solve.

Seriously love this product. What the Realm guys are missing is the fact that Couchbase Lite is really a replication protocol overlayed SQLite. Couchbase Lite did not always use SQLite and is in the process of switching to ForestDB as the underlying persistent store. As far as API's go, I have not seen a more intuitive design than CBL's. It almost makes too much sense. Once you get Sync Gateway setup, you have a mini iCloud. Watch the CloudKit presentations and you'll see that CBLite solves what Apple is leaving up to developers as far as Sync goes.

There is another produce coming out from Cloudant very similar, but with even more friendly api IMHO
https://cloudant.com/product/cloudant-features/sync/

I think that's because Cloudant's solution is the same replication protocol (CouchDB) talking to it's hosted databases. Thank's for pointing this out, I didn't know about.

CoreData is not as stable as advertised. If you use it simply it works. However advanced use will quickly discover sharp edges. I devote a non-trivial amount of code managing objects between contexts and there are still FRC bugs. (e.g. sorted update.)
And it's all closed source.

To me the more important part is the interface and thread safety.

Looks interesting! The Swift examples on the homepage seem to be showing Obj-C code: http://cl.ly/image/2r250h3a0A1y

Sorry about that. We’ve been seeing some issues with our CloudFront caches all morning, but if you give it a hard reload or two in your browser, you should eventually see our Swift samples — we’ll keep working to fix the issue.

I would love to share the swift-specific portion of the getting started docs on www.sososwift.com but http://realm.io/docs/ios/0.80.0/#swift doesn't navigate directly to that. If there's a link to Swift instructions for realm, I'll post them and you will find them here http://www.sososwift.com/?a=realm.io Good luck, looks cool!

CloudFront is giving us a lot of issues at the moment, but I implemented + deployed that feature for you! Hopefully it will be live very soon. (btw, thanks for creating & running SoSoSwift!)

Hey great, it worked! Sorry for the delay, I had to work at my actual job. :/ BTW, we just deployed a new app last week, had a bunch of problems with CloudFront. Good luck!

Sounds interesting. I'll definitely watch for the Android version :)
That said, the "insert" benchmark source for SQLite is a bit disingenuous. It makes a new compiled statement every time it inserts - if you're doing 150,000 inserts, you should definitely be reusing that thing. And on Android, greenDAO (probably others) does this automatically for every insert/update. It's not some edge-casey optimization trick, it's normal SQLite use.

Thanks for the feedback, benchmarks are notoriously difficult to get accurate (and fair), which is exactly why we published the benchmark code. We will incorporate your feedback to improve them.

I'm a little late to the party, but I ran the benchmark inserts with cached statements on both my Macbook Pro and my iPhone 5S. sqlite definitely fairs better under this scenario.
That being said I still find the speed of Realm pretty impressive. And it's hard to gauge real world performance from this test. In my experience the real bottle necks around insert are upsets and establishing maintaining relationships.
For reference here are the benchmarks without reusing sqlite statements.
Macbook Pro - Realm: 0.780414 sqlite: 0.464804 FMDB: 0.654659 iPhone 5S - Realm: 2.235381 sqlite: 2.786381 FMDB: 3.336679
And with statement reuse
Macbook Pro - Realm: 0.766406 sqlite: 0.189747 FMDB: 0.683823 iPhone 5S - Realm: 2.214847 sqlite: 0.991864 FMDB: 2.832800
Interesting side note: FMDB seems to be spending just as much time finding the cached statement as sqlite takes to prepare a new one. I didn't expect that. I haven't dug into it at all, but I wouldn't be surprised if this due to the fact that the insert statement is relatively simple.
EDIT: Just fixed up formatting for the results

That's pretty cool, and roughly what I saw in my emulator - maybe my fumbling actually did it right :) And very much agreed, Realm is pretty impressive, and micro-benchmarks like this aren't all that useful.
Have you set up a pull request / issue by any chance? It paints Realm in a bit worse of a light, but hopefully they'll merge it in and update the site.

I didn't see the benchmark code in the repo, so I didn't really think about a pull request. Next time I find time I'll poke around a bit more. They are trivial changes though.

Great! I don't know enough iOS-SQLite to confidently tweak it myself, and my small experiment (in an emulator, so ginormous caveats apply) gave mostly-unbelievable results so I somewhat doubt I did it right.
I'll definitely watch with interest! I'd love a safe, faster, easier data store than SQLite. I'm certain such a thing is possible, and it would address a massive range of uses. Good luck though - SQLite is pretty fantastic, it's a tough competitor to have :)

Should I submit an Issue on github to help push this, or is it relatively certain that it'll get done? If I get time + a device provisioning profile, I could also try to do it myself, but that's not likely to be any time soon :)

Awesome, I'm really happy to see this kind of technology. This stuff needs to exist, and is really the future of the cloud IMHO.

Very cool. What's the thinking behind shipping with Swift support despite it still being in beta?

JP from Realm here, I worked on our (experimental) Swift support. Swift as a language is pretty unstable right now, but it's clear that Apple sees it as the future of iOS and OSX development. So we wanted to start experimenting with how to make Realm feel right at home in Swift.

Well, we have a local database now. Now we need a terminal for the phone and a text editor.

Cool idea. I had the same problem with the intro video that others have mentioned. I'm gonna read more when the Android version is out, since this might actually be able to fix a common pain point I have with my apps with data persistence (I'd like to be able to read recently touched questions in my Stack Exchange app even when I don't have internet, which requires a damn complex caching layer).
Just a side note: The SO link you have in the footer seems to not have any questions about your actual product in the tag, but about Java realms instead.

Yup, offline mode is definitely a big area of focus for us. Regarding the tag, I’m not sure what the protocol is for these things, but since it wasn’t used for a clear product/project/topic, we figured it was as good a destination as any, since we’d prefer to let everybody benefit from answers we may give to support questions, via SO.
I’m drafting something to dive deeper into the video now; anything in particular you’d like us to dive into?

I'd love deeper info about "we don’t require deserialization to get objects out of the file", but that might be somewhere in the documentation. I'm assuming you're faster than SQLite, so a demo showing those side by side might be a good selling point too.

Realm is not an ORM, but rather an integral part of your language. One key difference is that it does not have to copy all values in the objects back and fourth as you read them and save them, as most ORM's does. So that gives a huge performance improvement.
There are some benchmarks in our intro blogpost[0]. So I can recommend reading that for a bit more background information. We will have more blog posts coming out soon about the technical details.
[0] http://realm.io/news/introducing-realm/

This is looking very good and it could lower the pain of developping mobile apps, especially for independent studios. Looking forward to learn more about the details of data syncing.

Data syncing is an obvious need and Realm has actually been designed for that from day one. The entire underlying core is build to seamlessly record changes and coordinate them with a backend.
It is not part of the initial launch as we want to nail down the local api's with the community first, but look out for announcements soon.

What kind of consistency do you expect to provide with this future syncing feature? I assume it will be eventually consistent. Is that right? How will conflicts be resolved?

Yes, on mobile sync really only make sense if also works when the device has spotty or no connectivity, so that naturally entails eventual consistency in some form.
We are not ready to into details about how our sync solution will work yet, but watch out for some announcements soon.

EnduroSync (https://orandolabs.com) does exactly the same thing, and it already has syncing. It's available for iOS and Android now, with more platform support coming.

Repost:
https://news.ycombinator.com/item?id=8037029

Well I think everyone agrees it looks like something pretty interesting but I couldn't find anywhere what's the easiest way of migrating from CoreData w/sqlite to Realm? Not sure if the tradeoff of rebuilding everything would be worth it for the increase in performance.

We will surely add some guides to address that need (and many others). In the coming time we would love to expand on the documentation and clarification so you can learn Realm as easily as possible. Feel free to ping us with anything needing specific attention in the support forum: https://groups.google.com/forum/#!forum/realm-users
As for the tradeoff, I guess that heavily depends on the specifics of your app. You are the only one who can judge that depending on your current app pain points, maturity and future needs.

This is awesome. Really interested to see how this can affect wearable device development and data.

That Four Square demo is quite amazing, I'm looking forward to give a try in near future.

A big plus for sqlite is that you can use it everywhere. For example, for a REST service instead of return a JSON I just return the sqlite database. Also, I can see the data in the desktop, use it on python, etc.
This could work elsewhere too? Be usable in python?

You can do the same thing with Realm. We actually have a couple of key benefits over SQLite there, which are that our file size is much smaller (usually about 50%), and we don’t require deserialization to get objects out of the file (Realm data can be streamed directly into memory as native objects in your language).
On the minus side, We don’t work from everywhere yet, but we have internal bindings for Python, Ruby, C#, Java, PHP and a few other languages. We focused on the iOS launch so far, but we do plan to wrap up those additional implementations and share them on GitHub.

Does this mean Realm would be usable for Xamarin-based apps in the near future?

Realm peeps, can you explain this: "The only restriction is that objects can be used only on the thread on which they were created"? Is this same as Core Data limitation or is there more to it?

Looks great. Really looking forward to the Android version too!

Looks good. Would really like to see Swift support. Also, the example showing integration with REST services in interesting, seems like you could do more with that.

Realm currently supports Swift, though we're actively working to improve our Swift APIs[0]. You can find Swift sample code throughout our docs[1], and in our Swift examples on GitHub[2].
We learnt a lot about Swift introspection and interacting with Swift objects from the Objective-C runtime. We'll share our findings in some upcoming blog posts, so stay tuned!
[0]: https://github.com/realm/realm-cocoa/pull/549 [1]: http://realm.io/docs [2]: https://github.com/realm/realm-cocoa/tree/master/examples/sw...

The speed, simplicity, and lack of headaches (threads, contexts, etc) really make this an interesting solution. Hope these guys/gals crush this!

If you could make this automatically sync to a backend database, that would be amazing. It would be great to have the hard bits solved for us.

Has any succeeded building this in rubymotion?

I don’t think anyone at Realm has ever tried, but I doubt it would work at this point. We do a lot of things specific to each language in the implementation, so we’d have to support RubyMotion explicitly for it work.

RubyMotion is Objective-C under some thin covers so it seems like it would be relatively straightforward to have it work with Realm without a lot of extra trouble.

I'll be following up with this a lot because I would like to cover it on MotionInMotion

I have to give it a try, but how well does it work with different BaaS providers like Parse, Kinvey, Appacitive etc?

Right now there is no specific integration into any BaaS.
A Realm database is just a (compact) file that you can send as you see fit. As most BaaS solutions has their own local storage and API it may not always make sense to integrate, except that you could easily store a small Realm file as a blob in any BaaS and transfer it remotely.
It will also really soon be possible to export some or all objects as json which you then again can send to remote services. That does however take up much more space/bandwidth compared to Realms compact data format, but could be needed if you want to persist remotely in another database format.

Not all of them generally have their own local storage. But as you do with the foursquare APIs, it can definitely be done with other API providers. But thanks.

Salmon pink? Their swag is going to rock.

There's a quote from Unity CEO on the frontpage — should we expect Realm appear in Unity-land as well?

Doubt it. It's pretty common to ask "testimonials" from people with authority for publicity etc.

How do you backup this database? Does it integrate with iCloud or whatever is available on the device?

The database is stored in a single file that is easily backed up. It does not have any explicit integration with iCloud, but you can easily use it to back up a copy of the db, or you can stream the copy back home yourself via ftp or http.

I'm somewhat of a newbie in regards to developing on mobile devices. Got any documentation how this is done from the phone?

I'm Kenneth at Realm. As realm databases are files on the phone's file system, you can use the common classes for manipulating them. To do a backup, the NSFileManager class might be useful (the copyItemAtURL method might be what you're looking for). See https://developer.apple.com/library/mac/documentation/Cocoa/... for details.

Is there a Mac OS X support planned? Or perhaps the iOS version works as is?

The framework in our release zip only works for iOS but if you build from source[0] it will actually generate an OSX framework as well!
[0] https://github.com/realm/realm-cocoa#building-realm

The RealmBrowser (which can be found under /tools/ at the above source link) is already using Realm internally to show/edit any Realm file dynamically.

I can't tell -- does it support upserts?

Currently not as a specific one liner. It's super simple to do it yourself, but we will likely add that convenience method shortly.

looks cool, can't wait to try this out!

Reply on news.ycombinator.com