Hacker News Reader

Comment by binarycrusader | Hacker News Reader

What terrifies me (as a MacBook Pro owner) is the recent realisation (due to some random corruption of iphoto data, and yes I have a backup device to restore from) that OS X has a fundamentally unreliable filesystem sitting on top of ever-expanding storage devices that many of us are depending on to archive our photos and other data.

I don't understand why Apple keeps pushing the capabilities of the system further and their hardware but haven't yet addressed the critical data storage issues that are eating away at the edges.

View on HN

As a former Apple Technical Support Rep of about 5 years ago iPhoto has always had a reliability issue. Back then once you approached 2GB it would begin going slow and corruption had a higher chance. It's slowly gotten better over the years but reading all of the internal support articles at Apple it was alarming just how many issues it had at the time.

Perhaps it's far better today, I'm not really sure but it completely turned me off to using it at all.

You realize you're commenting on a post about Apple's brand new app Photos, which is not the same thing as iPhoto, right? It's a replacement, sure, but it's a new codebase rather than an update to iPhoto.

> You realize you're commenting on a post about Apple's brand new app Photos, which is not the same thing as iPhoto, right?

I was commenting on a comment which talked about Apple's file system in a critical manner discussing how it can become corrupted. I shared a story regarding how iPhoto had many issues like this years ago which makes me wary of using any Apple photo application. Seemed on topic with the thread to me.

> it's a new codebase rather than an update to iPhoto.

Source? I had not heard this. I heard multiple things mostly which pointed to basing the new application off of Aperture's code base.

The point of your comment seems to be "iPhoto is untrustworthy, I don't think I'm willing to give it another shot". Photos.app isn't iPhoto. I think I was perfectly warranted in pointing that out.

> I heard multiple things mostly which pointed to basing the new application off of Aperture's code base.

I actually hadn't heard that, but I guess what I meant was Photos.app isn't based on the iPhoto codebase. Whether it's 100% fresh code, or whether it's based on the Aperture codebase, I don't know.

> The point of your comment seems to be "iPhoto is untrustworthy, I don't think I'm willing to give it another shot". Photos.app isn't iPhoto. I think I was perfectly warranted in pointing that out.

So you decided to completely ignore my explanation as to what I was commenting on and simply reiterated what you already said? Why troll HN? I don't get it.

For something that foundational, you have to move as one giant, lumbering beast. Recall that Apple had at least one (that we know of) false start. A fair bit of the OS (CoreServices; Carbon, and the apps that still rely on it) still relies on HFS+-specific data structures and features.

Core Storage is already moving forward as an effort to dissociate the bytes in logical storage from the bits on physical media. In the short term, we might be able to sandbox the evils of HFS+ by having it just be a software abstraction on top of a much more modern file-system-like-thing. Yosemite already moves any machine it upgrades to Core Storage, even if you don't use FileVault.

Plus, since the day they added journaling to HFS+, I haven't lost a single byte of data due to corruption of any kind.

I've had drives failed, where I lost a day's data (Was traveling and away from my time machine drive) and that sucked.

But near as I can tell, beyond losing short periods of work due to catastrophic failures of drives (early SSDs were problematic!) I haven't lost any data since Journalling was added (of course, Time Machine has saved my bacon from drive failures.)

I think Apple has done a really great job in this regard. Yes, HFS+ is based on filesystem work going back to the original Macintosh, but in use it's working fine.

I've had HFS+ corrrupt my TimeMachine partition, and Disk Utility was unable to repair it. DiskWarroir was no use as the volume was too big (the new 64-bit version addresses this).

HFS+ is still unreliable. The journalling is only for file system meta data. The data in the files is not journaled or checksumed, and can be corrupted without detection.

I experience HFS+ corruption frequently on my TimeMachine disk. There's nothing wrong with with the disk, it's the file system causing it. Disk Utiliy is usually able to repair it, but I did loose some backups once.

How are you verifying that you've never lost a byte of data? What makes you so sure?

Literally the first feature listed is iCloud Photo Library, a solution to critical photo data storage. It backs it up offsite which is better protection than even the best filesystem software.

If I'm offline for some period of time and upload photos to my laptop, an unreliable filesystem has every chance to corrupt my data before it gets uploaded to the cloud when I eventually reconnect.

The "cloud" option itself simplifies the backup process and could potentially reduce the window of a chance for corruption, but it does not eliminate the need for a reliable, local filesystem.

Wouldn't you still have the copy on the device or memory card you took the photo from as an alternate source to restore from? Just avoid wiping that until you have two other copies.

If the device is an iPhone[0], then there is only one copy on one filesystem on one memory card.

Granted, if your iPhone is offline for extended periods of time, you're also doing it wrong. But maybe you have an iPod Touch instead.

[0] For the last few years, the newest iPhone is the #1 most common camera model found in EXIF data on Flickr, http://petapixel.com/2015/01/09/popular-cameras-flickr-2014/. IIRC, Facebook saw similar results for photo uploads.

This seems terrifying if the filesystem is that bad that you have to do this to guard against corruption. And isn't half the point of offloading images from a card to clear up the card for reuse?

I don't clear photos off SD Cards until the files are in my offsite backup.

I don't accept the premise that filesystems are "unreliable" because you experienced a case of data loss. Hard drives can and regularly do experience hardware failures even with the best possible filesystem software. Camera memory cards can experience data loss before you get around to uploading. Reliability is like security; it's a matter of degrees not an absolute measure.

That is why a good, well-integrated offsite backup solution that "reduces the window of a chance for corruption" is a much more significant impact on data security than anything we could hope to incrementally improve in the filesystem software layer.

And Apple's solution is even better than you describe. Photos taken from Apple cameras (which are the vast bulk of photos for most customers) are automatically uploaded directly to the cloud as soon as the device joins wifi.

And you seem to be completely ignoring the fact that a filesystem that can't even guarantee that it wrote the data to an intermediate storage device correctly is fundamentally unreliable.

That is the core issue.

Yes, well-integrated offline backup solutions mitigate some aspects of the problem, but when your filesystem can't even guarantee that it did its job correctly, it doesn't matter how well-integrated your "backup" solutions are because the data may already be corrupt.

Nevermind that the "well-integrated" backup solution you are describing here, because of the unreliable filesystem, can't even guarantee that it read the data from disk correctly to a reasonable degree.

This is not some anecdotal "Mac OS X burned me" story -- this is a fundamental, architectural issue. Data loss is real -- and so are my concerns about HFS+.

Why do you think HFS+ can't guarantee that it wrote data to a storage device? From what I know (F_FULLFSYNC, etc) it does have that capability.

And why isn't he giving more details? He's being kinda broad without being specific enough.

At this point my guess is he's been running HFS+ without journaling turned on (you can format drives this way) and experienced data loss as a result because the filesystems ability to correct itself was disabled.

This would look like what he's describing.

No, I'm using the defaults for what OS X specifies when you first install. Which means journaling is turned on.

See John Siracusa's article:

http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/

Guarantee that it wrote the data correctly. Think checksums, like what ZFS/Btrfs have:

http://en.wikipedia.org/wiki/Data_corruption#SILENT

Same goes for when it reads back the data.

I'm sorry to hear you lost data due to that issue.

NTFS, ext4, and other consumer-targeted filesystems do not have this feature either, so it's unfair to imply this is somehow specific to HFS+ or Macs. You'd have the same issue on a Windows or Ubuntu install.

Actually, as of Windows 8.1 (64-bit only), ReFS is now supported which does have those features:

https://en.wikipedia.org/wiki/ReFS

So in the near future, Windows will have an answer for the general mainstream, while OS X will be left far behind.

Even ignoring the checksumming issue, HFS+ has known pathologies not only in reliability but in performance compared to NTFS or even ext4 on Linux.

HFS+ is long overdue for replacement.

No, such services are not proper backups. Their server software makes an error, syncs with your iPhone and Mac. Poof, your photos are gone.

For proper backups, backup to a local external disk (e.g. a time capsule or NAS) and an externally with incremental backups (e.g. Arq, Backblaze, tarsnap).

I use Arq, and the nice thing (in contrast to my Time Machine backups) data is also backed up on the go.

For proper backups, backup to a local external disk

Is a backup really a backup if it's onsite? You don't want a fire to take out your PC and external disk located in the same house.

That's why you use an external backup as well. It's just quicker to restore your data if your machine (or cloud service) failed, but your backup disk is fine.

That is a belt-and-suspenders approach to backups, but sure, it's a little safer.

However if it's a choice between onsite and offsite backups, which practically speaking it is for most people, definitely offsite wins. The risk of fire, theft, etc. is vastly greater than the risk that Apple has a critical syncing bug and no ability to recover lost data. (Normally they keep deleted files around for 30 days.)

Yes, but it also happens that someone deletes a photo from a stream accidentally and does not realize within 30 days. For this reason, iCloud Photo Stream is not a backup - a backup keeps copies until the end of time.

I've lost data (well, they were backed up, so not really lost...) to playing with beta filesystems (reiserfs 4, if memory serves well), never to HFS+.

I'm pretty sure somewhere in Cupertino a team is working on a new file-system, but I seriously hope they don't ship it before it's ready. I wouldn't mind playing with it in beta though...

Yeah, I'd heard that with reiserfs your data could just disappear one day without a trace. Once I got the developer to admit fault he was able to show me where he'd hidden the data. It was badly corrupted by then but still identifiable.

>that OS X has a fundamentally unreliable filesystem

Let's put it this way: OS X's filesystem is in no way "fundamentally unreliable".

HFS+ is not fundamentally unreliable. The potential exists for data corruption to occur in the amount of time required for the kernel to panic and halt. Even when that happens it still requires the stars to align and catch the HFS+ process writing data to the wrong place with exactly the right bits of memory being flipped. Even when that happens the chances or those bits landing on non-free user data with no file format supplied error correction further reduces the chances of data corruption. Even when that happens many file formats suffering from data corruption can be salvaged. Of course if you have a backup of any kind the odds of the same exact file / bits on both drives being corrupted causing permanent data loss are astronomical. Add a third offsite / cloud backup and it'd be way more worthwhile to worry about being eaten by sharks.

As the great teacher of my Monte Carlo course taught us, "If something can happen, it will."

It's not fundamentally reliable either.

From: http://en.wikipedia.org/wiki/Hierarchical_File_System "The Catalog File, which stores all the file and directory records in a single data structure, results in performance problems when the system allows multitasking, as only one program can write to this structure at a time, meaning that many programs may be waiting in queue due to one program "hogging" the system.[2] It is also a serious reliability concern, as damage to this file can destroy the entire file system."

I don't know much about filesystems. How can a filesystem have higher chance of corrupting data? And what is different about HFS that makes it likely to corrupt data?

It's got Journalling, and since that point I've seen no evidence of it corrupting any data, despite several catastrophic drive failures (early SSDs) and the need to restore from Time Machine.

"fundamentally unreliable"

I don't know if I buy that. But I do agree with you that Apple needs to improve its file system offerings.

HFS is a terrible fs design. HFS+ marginally improves it.

Consider that in HFS (and early HFS+) had a fun limitation -- you could only ever create 65535 files. Didn't matter if you deleted; the Catalog (CNID) incremented with each new file created until it ran out.

Also HFS+ only has 1 second resolution for timestamps and can't do timestamps later than February 6, 2040.

Other dreadful details here: http://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/

Get a NAS...

I've got a NAS but these things come with gigabit Ethernet and Marvel processors that will no way be able to cope with gigabit speeds!

Doing a Time Machine backup to NAS isn't fun, I can tell you. Particularly if you use VMs (which you can exclude, but why would you when it's an entire machine image eh?)

HP's Microservers can cope with gigabit speeds and the AMD models are dirt cheap, both compared to poorly engineered "NAS" devices, and especially for what they can provide in terms of server level features.

I've had the older N36L model, an AMD Athlon NEO II processor with 8GB ECC RAM running for years with ZFS handling a few big spinning drives for storage, it's been quite nice and trouble free.

That beats my piece-of-junk NAS then! Thanks Freecom! USB3 and Gigabit Ethernet (and an always open SSH port 22 yet no responsive server on that address...)

This is my solution, but honestly there's a lot to be desired, especially with regard to photos.

Storing Aperture, iPhoto, or Lightroom databases on the network is not ideal. Wifi, even 802.11ac, is barely usable. Gigabit means you're wired and also has issues. Apple frequently has implementation bugs on both AFP and SMB which slow things down way below gigabit speeds, and your NAS's processor also becomes a factor. Forget about using this setup if you're not actually on the local network (i.e. VPN).

The result is having to manually manage your photo databases (which one's are on my laptop? which ones are on my NAS?) which works ok in Lightroom but Aperture and iPhoto are not very good at. This also creates backup stress (um, can I delete this?) and means I have to decide in advance if I want to look at those photos from two weeks ago before I leave the house. If so, grab a cup of coffee while they copy.

Ideally you would have a system where a) your photos are automatically stored in the cloud b) you can select which albums are stored locally and c) there is some sort of cloud backup option that you control (computer with more storage, NAS, or other cloud provider that archives your cloud data).

> OS X has a fundamentally unreliable filesystem

Unfortunately so is every other filesystem. Which is also why Apple heavily emphasizes making backups.

Have you looked at ZFS?

Have YOU looked at ZFS, and how it's unfit for desktop use?

That's unsubstantiated FUD.

You only have to glance through this thread to see that there are people (myself included) who run ZFS on their desktop machine (laptop in my case).

People love to over exaggerate the RAM requirements for ZFS because they see build logs of enterprise-grade storage servers. But you wouldn't expect that kind of throughput on your desktop systems even with lighter file systems. And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead.

Obviously the real crux of argument is "what do you primarily use your desktop for"? If it's gaming, then there's little point running ZFS since you're going to be disinterested in the benefits of ZFS anyway, so it's not worth the marginal overhead. However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option.

>And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead.

Most "desktops" today are in fact laptops, and the reason you want to have the CPU sit idle is battery time.

>However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option.

If you're not pooling disks together but just have the built in disk, and usb hard disks that you conenct occasionally you don't get much if anything from ZFS, but you get the penalties it incurs...

And it's not a substitute for backing up your disks either...

> Most "desktops" today are in fact laptops, and the reason you want to have the CPU sit idle is battery time.

That's a fair point. I'd argue that ZFS doesn't chuck that much extra CPU time that most people would notice. However if you're one of those guys that likes to tweak your power settings to the nth degree, then ZFS probably isn't best. But those kinds of people are more likely to run a tiling WM instead of a full fat compositing desktop environment; and such like. So they're definitely not your regular desktop users

> If you're not pooling disks together but just have the built in disk, and usb hard disks that you conenct occasionally you don't get much if anything from ZFS, but you get the penalties it incurs...

You seem to have missed quite a number of useful ZFS features off your list. Ignoring deduplication (as that is extremely memory and CPU hungry), ZFS has snapshotting, checksumming, superblock recovery (even if your journal gets trashed, you can recover it), online data integrity checks (no annoying wait screens for fsck to run through), better compression algorithms than is supported in most other file systems...

I could go on, but suffice to say that ZFS is an improvement over many other file systems in other areas than just software RAIDing.

That all said, in an ideal world desktops would be running HAMMER (DragonflyBSD's file system) over ZFS. I do love ZFS, but HAMMER seems a slightly better fit for desktops / laptops as it has the aforementioned features but without many of the data centre stuff that home users wouldn't need. A better compromise. However I don't see much interest for HAMMER outside of Dragonfly.

> And it's not a substitute for backing up your disks either...

Nobody suggested it was. However ZFS does also have backup features built into it too.

Yea but there is a continuum of reliability -- ZFS is way better than HFS+ in this regard. Also, Btrfs.

I've used btrfs (at several-petabytes-of-data scale) and wouldn't trust it any more than HFS+.

Use ZFS if you like, or XFS, or even ext4 instead.

Would love to hear more about your findings with btrfs.

The biggest recurring issue was deadlocks in the garbage collector. It would start cleanup in a subvolume and trip all over itself. After that, any I/O to that specific directory would never return. The only solutions was to reboot the server and fsck for a few hours.

Second frequent problem: Hitting 90% capacity in a filesystem has a non-trivial chance to ruin it forever. Hit the wrong code path, and, even if you immediately delete a bunch of things, I/O to that filesystem would be forever 3000% slower.

I would also like to learn more about your view of btrfs. Is it a fundamental problem? Or an implementation, or community issue you have?

What are you talking about? HFS is creaky and ancient and doesn't have checksums, ZFS and btrfs do.

The way I see it:

HFS+ is creaky, ancient, but battle-hardened.

ZFS is solid and battle-hardened, but not that suitable for many of Apple's customers, who want the ability to unmount disks they added, and who will not understand that they need huge amounts of RAM (they their Macoboks don't have) to keep their disks speedy,

btrfs is promising but not battle-hardened yet, and it is very unlikely Apple will use it as long as it has a GPL license (I think it is more likely that Apple will add checksums to HFS+ in some somewhat clumsy, but working, way than that it starts supporting btrfs)

> HFS+ is creaky, ancient, but battle-hardened.

If you think HFS and HFS+ are battle-hardened I don't know what to say to you. "Shell-shocked" would be a better description.

Tens of millions of users have been using HFS for decades mostly with zero issues. I personally have used a Mac since the Mac+ and not one had corruption due to the filesystem. I would describe it as battle hardened for sure.

> I personally have used a Mac since the Mac+ and not one had corruption due to the filesystem.

You're a very lucky man, then. A large part of my time, at one point in my life, was spent fixing up corrupted HFS filesystems.

“mostly with zero issues” is the same as saying “with issues”.

I would generally agree that it's rare for e.g. HFS to corrupt its own metadata due to bugs but bit-rot happens. If you think you haven't encountered it, you probably haven't actually checked your files closely enough – this is particularly easy to miss with things like video or image formats which were designed to tolerate a few bit-flips.

The reason why so many Mac users were interested in ZFS was that a modern filesystem incorporates strong integrity checks so you are guaranteed to either read back the bytes you originally wrote or get an error. That's a much better way to work than requiring you to use a tool which does strong hashes periodically to see if anything has broken.

Something like ZFS would be a double edge sword for Apple.

A lot of consumers are quite obvlious to bit rot. As you point out - most of this occurs in large media files which they are unlikely to notice. Even with things like corrupted documents they will likely blame it on an application glitch.

With ZFS they will see scary diagnostic messages. Consumers will want their SSD or RAM replaced. (ZFS does a great job of catching RAM errors on systems without ECC)

This would no doubt drive higher warranty costs for Apple and cause consumer dissatisfaction.

Ignore is bliss...

> This would no doubt drive higher warranty costs for Apple and cause consumer dissatisfaction.

I'm skeptical that a proactive warning would be more dissatisfying than just finding it out years later, when other copies are harder to locate. In particular, I'd be surprised if a hypothetical feature like this wouldn't be integrated with their other services: “A local failure was detected and replaced with a pristine copy from iCloud”

The other really interesting question would be whether they'd ever consider something like the ZFS copies feature to directly trade maximum disk capacity for redundancy. That'd be much easier to implement for Apple since they control so much of the platform and the largest files which most people have tend to be things which are provably known to be local copies of something which can be re-downloaded, so the total cost of doubling storage for unique local files might be worth the peace of mind for many users.

> This would no doubt drive higher warranty costs for Apple and cause consumer dissatisfaction.

I suspect the decision not to use ZFS was (assuming it wasn't simply spite at Sun pre-empting Jobs' announcement) that it would cause dissatisfaction, but from a different direction: ZFS was designed for Sun's customers: companies employing dedicated sysadmins. ZFS was incredibly unfriendly back when Apple would have been evaluating it. User experiences like "you filled up your filesystem? Oh well, you'll have to reformat and restore from backup." may be acceptable in an enterprisey setting, but aren't going to cut it with the iMac crowd.

>“mostly with zero issues” is the same as saying “with issues”.

In the same logic people had issues with ZFS too.

"With issues" means nothing. The number and severity of them only has meaning, and even that only comparatevely.

FAT is creaky, ancient, and battle-hardened. It's the only filesystem that all OSes can use properly and can be used as a sort of rosetta stone as a result. It's still not suitable for data storage.

And yet again hundreds of millions of people trust their photos to be stored on FAT formatted memory cards.

Sorry, I should have said 'long term' data storage. It's frequently used for short term/sneakernet sort of thing.

… and every year, millions of people lose data because it was stored on a non-redundant filesystem without strong integrity checks.

There are multiple businesses which sell software which exhaustively attempts to recover data from corrupt drives & cards.

It's a standard service listed on the local phone / computer shops.

I see messages on my neighborhood's mailing list every so often from someone looking for help recovering as much as possible from a flash card.

It's simply absurd to claim that the status quo works for the general public. It's not terrible, probably not that different from various mishaps which befell printed photo albums, etc. but there's still plenty of room for the industry to give people safer tools.

There's certainly room for improvement but when the odds of file system / medium causing data loss are lower than more practical concerns like fires, theft, etc isn't that really good enough?

I don't know about btrfs but didn't you need the huge RAM amount for deduplication? Without that, ZFS would be serviceable on any reasonably recent Mac, except probably the new Macbook, or they would need, at most, an upgrade to 8GB.

Also, why wouldn't you be able to unmount a ZFS disk?

You can unmount ZFS storage really easily:

    zfs unmount ...

Or using umount like you would with any other file system.

Generally though, you wouldn't want to manually unmount file systems unless you're planning to migrate them to a new host. In which case you would export them instead:

    zpool export ...

A backup doesn't help you if the photo is corrupted, and then that corrupted photo is propagated to your backups. You need a way to catch file corruption, like what ZFS and Btrfs have.

btrfs still isn't 'production ready', unfortunately.

ZFS is, and Apple almost had it. Damn.

ZFS is totally broken for regular desktop use. It's use case is a different thing altogether.

Do people just read the "pros" and not the "cons"?

Have you tried https://openzfsonosx.org ? I use it daily on my primary desktop for my home directory and with an external time machine backup drive and have had zero problems.

ZFS doesn't have a specific "use case". In fact a large amount of effort went into ensuring that ZFS was tolerant of consumer hardware, so that makes it as sensible option for desktop use as ext4 or HFS+ (when looking strictly at "use cases").

You've also failed to quantify why ZFS is "totally broken for regular desktop use" (likely because you've never actually used ZFS so can only blow smoke about it's issues).

Obviously the real crux of argument is "what do you primarily use your desktop for"? If it's gaming, then there's little point running ZFS since you're going to be disinterested in the benefits of ZFS (plus likely running Windows anyway). However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option.

I've head John Siracusa discuss this, but I can't find a good link at the moment. Yes, with cloud storage systems, there is a chance that with HFS+ you're just backing up garbage and possibly won't notice until you use the file. HFS+ is not really equivalent to its peers.

Here you are, it's from the Lion review I think. Pretty scary reading, and particularly since we've had numerous releases since then with not many changes:

http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/

Reply on news.ycombinator.com