Why iOS Devs Should Consider Using Core Data iCloud sync
zaal.tumblr.com
zaal.tumblr.com
iCloud can resolve this problem by letting your users edit data offline and
resolving any merge conflicts automatically (and in the case of iCloud Document
syncing, giving you some options to get feedback from the user to help resolve
the merge conflicts). If this worked seamlessly, it would add great value to
your product.
App Store Features: you might get featured by Apple on the App Store for using
their latest, greatest technologies.
Future Plans to move to Android / Web: a legitimate concern. But why not wait
till your iOS product has grown big enough to have substantial demand for
Android and/or Web, and develop your custom back-end solution then?
- using Apple-sanctioned technologies might improve your chances of getting featured by Apple. Of course, well-known popular apps get featured regularly by Apple, but this is one of the factors that goes into their calculations of who to feature. This is not a reason per se for using iCloud, but an added incentive (as I mention, it really didn't pan out for me!)
- Exporting data out of iCloud shouldn't be any more complicated than importing data into iCloud is. You can easily walk through the Core Data object graph and do whatever you want with the data. And with NSIncrementalStore, you can easily keep the core of your application the same and utilize some other web service to power the data for your app.
My team at Couchbase has been building mobile NoSQL databases that sync[1], for a number of years now, and our tech is used by a few apps in the app store already.[2]
This preamble is to say that I've had plenty of conversations with developers who would be in iCloud's target market but decided not to use it. And the big reason they won't use it has very little to do with the reliability and user experience problems that keep coming up in the critiques of iCloud. The big reason they don't use it, is that they need to control their data.
It's one thing for a small game or journaling app to outsource data management to Apple, but an entirely different thing for medical apps, enterprise sales, or other "serious" apps to do that. Perhaps in the long term Apple will offer an enterprise version of iCloud that you can run on your own terms. But until that happens, most serious apps are going to want to run their own infrastructure.
BTW all the code we write is open source, so if you are interested in sync capable databases with offline and p2p support, join us. [3]
[1] https://github.com/couchbase/couchbase-lite-ios [2] https://github.com/couchbaselabs/TouchDB-iOS/wiki/TouchDB-In... [3] https://groups.google.com/forum/#!forum/mobile-couchbase
> "Yes, there are various problems"
Yes, chief among them that it doesn't work. I'm quite serious here. No one has gotten an iCloud-based Core Data store to withstand the actual multi-device usage it was designed for.
We're not talking about "works, but buggy". We're not even talking about "works but is a pain in the ass". We're talking "core features do not function". We're talking about "no known workarounds to bugs at the core of system that prevent it from working in any way that resembles the documentation".
And it hasn't worked since iOS 5.0, released almost two years ago. We have been through another major yearly update and several point updates, and no visible progress has been made on Core Data iCloud sync. The documentation has not improved, and with every point release developers go back and survey the situation only to find more bugs and more and different obscure error codes being emitted by the system.
Core Data iCloud sync would be bearable if it were just buggy. It would even be bearable if there was forward momentum on the vast amount of brokenness in it. But neither have demonstrated themselves.
But enough whining, let's address some points made by the article.
> "Or you can allow offline editing and build your own custom conflict-resolution solution, which I can guarantee will make YOU very unhappy... iCloud can resolve this problem by letting your users edit data offline and resolving any merge conflicts automatically"
Yes indeed, why build your own conflict-resolution layer, which you will probably get wrong in horrible ways, when you can rely on a working solution?
Poignant question if there was a working solution. Author is describing an iCloud that does not exist. Core Data iCloud Sync conflict-resolution does not work, and when it does not work, it has failure modes which result in the irrecoverable corruption of your entire data store, and in combination with other iCloud bugs results in the complete inability for your app to edit its iCloud bucket ever again.
Ever again.
I wish I didn't have to use so much italics here, but the point really must be made.
> "even if your app has been terminated, the iCloud daemon can keep running and upload your data whenever it gets an opportunity"
> "with CDIS, data is pushed into your app when it’s ready"
Indeed, these are the points for using iCloud over other competing cloud sync services on iOS.
But that's a little like saying the Ford Pinto has a really nice tape deck. It has this really neat, really useful feature when it isn't exploding. It's like saying dinner service on the Titanic is great - none of this matters when you can't even get the boat to float.
> "And that they have really seasoned, very smart people working on that Core Data team. I can’t imagine Apple releasing something as ambitious as a distributed, decentralized database system without “thinking it through”."
But they didn't think it through.
Author, you've been working with CDIS for the last two years. The technology behind it is neither mysterious nor is it secret. CDIS isn't broken because Apple can't write code. CDIS is broken because it is architecturally unsound.
Core Data iCloud Sync breaks down from a high level like this:
- You break each transaction into your database out as a diff. You store this diff in a directory that the iCloud daemon is syncing. Thus, all new diffs (called transaction logs by CDIS) are replicated to all devices. So far so good.
- It is trivially easy for a combination of devices (or really, just two) to generate a combination of diffs that do not playback to a consistent database state.
- In a normal world where CRUD operations are handled by a server with some understanding of the underlying data model, the server resolves conflicts as the authoritative data source, and thus consistency is maintained.
- In the world of CDIS, there is no authoritative server, nor is one device ever designated as the canonical state (this would be unrealistic, since devices get retired and lost, nor is there anything special about a device that makes it canonical). Instead, conflict resolution is left to each device, the implementation of which is broken. Which is to say, your CDIS-enabled database will work fine until the moment two devices make conflicting changes. At which point it is irrecoverably corrupted.
- The CDIS client's reaction to this is to revert your local copy of the database to its last known good state and cease communicating with iCloud entirely. This error occurs silently without either notification via UI (to the user) or API (to the developer). There is no way to query this state, and as of today the only visibility into this error is via console logging.
- Because the corruption has already occurred server-side, this means all connected devices using this store are all now disconnected from iCloud when using the app.
- As previously mentioned, there is no simple way to detect this error state via the API. Even if you were able to figure out that this has occurred, there is no way to destroy your database and start over. Due to bugs, lack of documentation, or architectural oversights, there is enough metadata about the database that the app cannot delete, that deletion of the store and its associated diffs do not put iCloud back to a clean state.
This is why people are giving CDIS a lot of shit on the internet. Nobody seriously believes Apple is incompetent at programming, but even a mild dive into the underlying technology on which CDIS is based shows that this entire architecture is unworkable. It's not that Apple can't get it right, it's that they've committed to an architecture that no one can get right.
As someone who has spent the last few years neck-deep in iOS, who makes a living writing iOS apps, I want iCloud to work right. It is in my interest to cheer on Apple's every move. But short of completely scrapping their existing architecture I do not see how it can be meaningfully improved (read: made to actually work).
> "CDIS is a really compelling technology for many iOS / Mac developers to adopt in their apps, and you should be seriously looking at iOS6 or iOS7 as the point to jump onto the CDIS train if it fits your needs"
NO. I for one would be ecstatic if CDIS works as advertised in iOS7, but under no circumstances should anyone consider using CDIS on an iOS6 device.
> "and it deserves less dismissiveness"
Dear author. We, the iOS developer community, have not acted with dismissiveness about CDIS. In fact we raced towards it with high hopes. We fought the scant documentation. We fought the bugs. We poured into the dev forums in droves. We helped each other and we shared our learnings.
We are angry now because we have tried. Nobody dismissed CDIS out of hand - the anger here comes from tens of thousands, if not hundreds of thousands, of collective man hours exhausted attacking this problem from every possible angle, and we are no further from where we were when CDIS first dropped in iOS5.
All of the points you make are valid in a universe where the technology you're talking about is functional. It would be insanely great to have a cloud-based data store that does conflict resolution for you, that updates in the background, that removes our responsibility for hosting it, that has no operational overhead on our part, and pushes instead of pulls. Indeed, those are all great reasons to use such a technology - if it existed.
But as it is no one can make a Core Data iCloud store stand up for more than an hour. And no one can recover a corrupt data iCloud database after it's fallen over. So unless you've figured out the solution to the above two problems, I'd kindly suggest that you stop treating the entire iOS developer base like ignorant, petulant children.
That was way longer and way angrier than I intended, but the article is really IMO nonsensical, and the author's tone that the iOS community has someone "dismissed" the technology or somehow not given it a fair shot, bothered me greatly.
They have a great foundation in Obj-C/Cocoa Touch but I'm very quickly losing confidence in their ability to move the platform forward.
Android is much less mature in many ways but seems to be under much more expert guidance lately, both architecturally and at the UI level.
1. Interact poorly with version control.
2. Scale badly with a lot of VCs.
3. Segues are untyped and are inadequate to express complex navigation schemes.
AutoLayout:
1. Excessively verbose normal syntax and excessively limited and cryptic and untyped shorthand syntax.
2. IB tools are woefully inadequate for editing.
3. Small adjustments to complex layouts in IB completely scramble the constraints.
4. Does nothing to make the 95% use case of nested grid layouts explicit and simple.
Seems like the guys who build iawriter did or maybe we're talking about something different
My point was mostly to discourage people from completely writing off the idea around CDIS, that you shouldn't work with it "even if it worked", and countering some of the incorrect assumptions that some people were making. I think there's great reasons to adopt it (theoretically), and (this might be personal opinion) I don't believe that they're too far from cracking it. I guess time will tell. I've figured out a few hacks to backup data, to recover from failed syncs, walk through users with cleaning up their iCloud containers and get devices to setup sync again. They're mostly hacks that shouldn't be required, but have gotten to the point where users mostly have a great experience with the app, and even if sync fails or they start seeing weird errors, I can recover their data and walk them through restoring their state. I've only seen 3-4 people who have lost access to their data completely, and that really sucked. But that's a risk anyone takes with any technology, home-made or provided by some other platform.
I am open to the idea that I'm maybe too optimistic about all this. But having spent a lot of time and energy on CDIS, and having my app in production supporting CDIS for a few months, I think it was valuable information to share. A lot of issues that devs might run into while testing don't actually pan out in real-world usage. People don't have their devices sitting open side-by-side, trying to edit data simulatenously. But in any case, my plan is to compile a list of all the issues I've experienced next, and maybe some of the hacks that I'm using to work around them, and do another write-up in the near future.
I simply have not seen this attitude at all on the internet. Everybody wants the promise of CDIS. All of the anger, frustration, and vitriol is around the broken promise of it, not the concept.
> "and (this might be personal opinion) but I don't believe that they're too far from cracking it."
That remains to be seen. There is a growing sentiment in the Apple dev community that Apple's QA quality is slipping, and slipping hard, and this has a large detrimental impact on developers.
I was recently at a conference where this "joke" was made multiple times. Whenever someone commented that "X might be fixed in iOS7" someone would pipe up "Great, then we can use it when iOS8 comes out."
The upgrade rate on iOS devices far outstrips Android, but even now, on the eve of iOS7, a good 15-25% of devices are still on iOS5. There is effectively a one-year gap between Apple coming out with new API, and being able to require the use of the API in your app (unless you feel like leaving up to a quarter of your userbase out).
With things like CDIS this one-year gap is effectively lengthened to TWO years. Apple comes out with something new that's broken enough to be unusable in production. We wait another full year for them to finish baking it. And then we wait another year for the install base to be wide enough to ship an app with it.
With CDIS this is coming up to be a FOUR YEAR GAP, assuming they make it production-stable in iOS7.
There is plenty of reasonable skepticism when it comes to doubting whether CDIS will be fixed at all, or be quietly swept under the carpet as a failed technology.
> "I've only seen 3-4 people who have lost access to their data completely"
In my experience data loss has never been an issue - loss of iCloud connectivity is. Which is to say, once a conflict resolution failure occurs, that device is now an island - no changes made there will be communicated to any other device, and no changes made elsewhere makes it to that device either. And there is no way to restore this connection that anyone is aware of it.
Data loss hasn't ever been a major problem here - when a breaking conflict resolution occurs Core Data's default behavior is to go all-local rather than delete data.
The inability to repair a broken iCloud link is 100x more aggravating than simply breaking, since it means having to tell a user that they are SOL. By far the greatest frustration I've encountered re: CDIS is exactly this - that once it breaks, it's permanently broken.
> "But in any case, my plan is to compile a list of all the issues I've experienced next, and maybe some of the hacks that I'm using to work around them, and do another write-up in the near future."
I'm looking forward to it, though I remain extremely skeptical that Core Data is production-ready.
I've now, collectively, been in rooms with literally thousands of angry CDIS developers. Very smart developers who have yet to be able to ship a single CDIS-enabled app because its stability is simply atrocious. I'm wondering how it is that your experience is so different from everyone else's - where you reportedly have CDIS working in a production app with few/no issues, while others can't even get their Core Data stores to stand up straight. That's a pretty huge gap in progress.
- I referenced the Brent Simmons article and the Marco / Siracusa podcast early on in my post. They're all highly-visible and prominent members of the Apple dev community, and this was clearly their stance that I was challenging.
- Regarding upgrading, you can mark out certain features to be available on the latest version of iOS while maintaining general backward compatibility with the previous iOS version. My app is compatible with iOS5, but I don't expose the iCloud sync option to iOS5 users because I don't want them to use it. I've had 0 complaints about this policy.
> The inability to repair a broken iCloud link is 100x more aggravating than simply breaking, since it means having to tell a user that they are SOL. By far the greatest frustration I've encountered re: CDIS is exactly this - that once it breaks, it's permanently broken.
This isn't true. I encounter this problem fairly regularly with users, and the process I outline for them is this:
- turn off iCloud sync option from my app from all devices. This will make an object-by-object copy of 'ubiquity' sqlite db into their local sandbox
- nuke their container from Settings -> iCloud -> Storage and Backup -> Manage Storage -> Documents and Data -> Show All -> <my app> -> Edit -> Delete All. This cleans out their iCloud database and all metadata associated with it
- wait a few minutes for the delete to sync over to all devices
- choose one of the devices as the most recent 'truth' version, and use that to be the first device to sync up to iCloud
- connect second device to iCloud again, and this will fetch data from iCloud instead of uploading. Once done,from then on, sync will work again seamlessly across both devices.
The instructions have some nuance in them, but that's the gist of it. Resetting the iCloud container fixes almost all the problems that you would encounter with CDIS, and the hacks are around making backups and restoration of data as painless as possible, and trying to minimize data loss. Again, this adds customer service add-on, but it's far far from impossible.
This is the part that's news to me - I've observed myself and corroborated with many other devs that nuking a container prevents the container from being recreated by that app ID ever again.
This really is the one big sticking point - if nuking containers actually worked properly the scope of the problem is greatly reduced. Sure, potential data loss, but at least your devices start talking again.
I'd be interested in seeing code, but this is definitely something outside of your app. Very curious. I was very recently in a room full of CDIS devs who complain that the above is still broken in iOS 6.1.3, and I'm at a loss as to how you've been able to achieve successful recreation of an iCloud container that has been deleted via Settings.
After nuking a store and waiting a few minutes (ideally restarting your device as well), the container is just as good as new. You can try it out with a basic CDIS app yourself and see.
I'm not calling you a liar - I'm quite confident it works fine for you, but I'm wondering how our experiences can be so different. This isn't just me, this is literally confirmed by entire room fulls of people who've been wrestling CDIS for months.
I'm baffled.
Occasionally, after doing this, it'll appear to take much longer to provision a new ubiquity store (sometimes minutes, which is unacceptable), but never actually broken.
For iOS it is pretty easy though, anything in Docs gets synced if logged in and the app is setup for iCloud. It has always been a little touchy when they added the Caches/Documents requirements. You have to setup a no backup flag to prevent it from syncing if storing anything there which can be a problem if you store too much.
BOOL success = [URL setResourceValue:[NSNumber numberWithBool:YES]
forKey:NSURLIsExcludedFromBackupKey error:&error];[1] http://www.ibm.com/developerworks/lotus/library/ls-NDHistory...
But iCloud is just badly implemented for the moment. Will probably get better. Apple doesn't call iCloud beta, and people will probably be pissed since it doesn't work properly.
I suspect that if they did call it Beta, some might have had gone easy on it for all the faults out there.
I gave it a try when the API was accessible, I didn't like it, never looked back.
1)SSO can work with other form of social logins - and will also work from other platforms/clients
2) Privacy? As a developer you have access to those files
3) This works better if it's stored on a server that has a more public API since you probably want a web interface and etc.
It's just a reference to check if your current device is participating correctly with the rest of the iCloud container, so that you can tell if someone deleted the iCloud container behind you while the app wasn't active, the app would be able to handle that situation. It's only exposed to the user in case something is wrong and the user can quickly check this value from that plist file. In practice, it hasn't proven too useful to expose this to users, but something to build on for a future update.