iOS 12: Core Data External Binary Data Storage Becomes Corrupt | Hacker News Reader

iOS 12: Core Data External Binary Data Storage Becomes Corrupt

mjtsai.com

149 pointsepaga33 comments

Golden rule: don't upgrade until first sevice pack. Release date set by management, so you have to ship with whatever bugs are left. But patch is usually developed with more timely approach.

I reported some bugs as early as the 2nd and 3rd Developer Previews of macOS Mojave, but they’re still not fixed in the final release.

I’ve been reporting the same bugs in Finder since Mavericks, don’t get your hopes up that they will be fixed.

Finder is the bug, unfortunately.

You’d think Apple would at least want to take care of any visual inconsistencies, specially in one of the major features which are touted as the big incentives for upgrading. There are still some UI elements here and there which are out of place in Dark Mode at best, or almost illegible at worst.

Such lax approach to UX polish has traditionally been associated with Microsoft, not Apple.

I reported (and had Apple engineers come to meet us) a severe bug (found and showed the engineers the single line of code in the public source code) in OSX 10.4’s wifi stack, wasn’t fixed until 10.8. Literally a single line of code.

That doesn't really work when the vast majority of iOS users upgrade so quickly. These devices mostly not in control of IT departments.

Yep. Just fyi, for those devices that are managed with an MDM solution, they've recently added the ability to delay upgrading for 90 days: https://appleinsider.com/articles/18/02/01/it-pros-will-be-a...

Largely agree but if everyone did this, bugs would not surface... for this to work, some people ought to upgrade and others hang back.. which is already what happens anyway...

Sure they would. You can avoid upgrading a device you depend on, while installing the latest beta and .0 OS releases on a test system or partition. That's what most developers I know do.

This particular functionality has been quite finicky since its beginning years ago.

A lot easier said than done when Apple stops signing old iOS versions immediately.

That just prevents you from downgrading after you already installed the update. There’s nothing stopping you from not updating to begin with.

You're assuming that nobody ever has to restore from backup or reset their phone. Reasonable people might disagree with you on this point.

Unfortunately that is true even with high quality engineering companies like Apple.

OK, I’m not an iOS developer. Can anyone explain what this means?

Core Data is an object persistence framework for iOS that essentially wraps around SQL. It looks like it’s losing data when the external storage option is enabled.

What does external storage refer to here? I don't think iOS has extendable storage does it? So is it eg. iCloud?

It appears to be "external" to the sqllite database

https://forums.raywenderlich.com/t/where-is-allows-external-...

If you have a large amount of data to store, instead of embedding it in-line inside the database itself you’d store it in a separate file.

Core Data is an object graph and persistence framework [1]

On iOS 12, binary data stored in Core Data with the External Storage option gets corrupted on every second save [2]

[1] https://en.wikipedia.org/wiki/Core_Data

[2] http://openradar.appspot.com/44884416

Core Data is basically a layer over SQLite that turns it into an object graph database.

External data storage lets you offload large binary data from being stored in the SQLite database without changing how you access that data from objects in the graph.

An example might be if you're creating a photo library. Rather than storing the actual photos as say BLOB data in SQLite, you can tell Core Data to transparently store it as an external file. It then basically stores the filename and some metadata in SQLite instead, keeping the SQLite database small without changing the Core Data API to access the data.

The Sqlite database becomes corrupt ?

I doubt it. The data is stored externally in the filesystem and referenced in the SQLite database. I guess something happens in the filesystem or during every second storing process, so that the metadata references become out of sync.

I'd guess it's because someone isn't accessing it from a single thread, or isn't following the lifecycle requirements for storing data in an asynchronous datastore.

You absolutely have to wait for the completion event for a write before issuing another write to the same key for example.

maybe they are using memory as a temp store and sync issues are popping up?

Nope.

CoreData has the option to store blobs as flat files, outside the SQLite DB. The DB only contains the metadata to locate the files. Looks like they're messing up these external files.

And I guess there's no way to create a transaction between the file system and the sqlite db. I've often thought that would be a useful feature sometimes - maybe something in the sqlite API could write your file, and then commit when verified on the filesystem.

I often promise myself to have a look at SQLite virtual file system API, for this very purpose, but could not yet find the guts :-)

That could have problems itself if the filesystem action succeeded but then the SQLite commit failed. Transactionality is hard!

What are the foundationDB guys doing ? After they got bought I was hoping so much that ios would get a brand new storage technology that really works well using their famous testing environment.

I don't know what's wrong with core data because throughout the years it's been one of those technology that are both unconvenient AND regularely unreliable.

FoundationDB on an embedded device? That's certainly an unorthodox approach, but I don't see the use case...

On why Core Data is inconvenient: it's as inconvenient as any ORM framework in general.

http://blogs.tedneward.com/post/the-vietnam-of-computer-scie...

Not the tech itself, but the people that built foundationDB surely know a thing or two about reliability and building a modern DB api.

Also, you're mentionning ORM, but relying on a relationnal database as the underlying storage layer for objects doesn't have to be.

Reply on news.ycombinator.com