Hacker News Reader

Comment by hosay123 | Hacker News Reader

I don't know the numbers, but the probability of getting 26 corrupted at-rest files through natural causes sounds pretty much like winning the lottery twice on the same day you were struck by lightning twice

Checksums wouldn't have fixed this, they'd only alert the user to the fact the damage had already been done, which is exactly what the decompressor did in its own special way.

As another comment points out, error correcting codes are the way to handle this, and its already done in hardware, and probably too expensive to do in software in the general case

View on HN

I think you're discounting the utility of simple checks.

Imagine you have weekly backups , but only for the past 26 weeks because of disk space. A file gets corrupted, but you only view it a year afterwards. You effectively have no way of recovering it.

Knowing something is wrong can be useful (though being able to fix it is even more useful).

That's exactly why I use par2 with all important data when backing it up. Bit corruption? So what?

You're right, that reported failure rate is way too low, I'd be glad to only have 26 corrupt files. I have one disk with about 300gb of CR2 files (HFS+). Not long ago I did a similar procedure as in the article (except that I also had the files on a ZFS mirror (I was cleaning up the HFS disk and checking for duplicates before formatting) and found 250 files with bit errors compared to the reference. Fortunately only about 1 in 3 of the bit errors resulted in corrupt files.

Needless to say I no longer keep anything of consequence on non-checksummed filesystems now.

It annoys me when someone says "you're right" and then completely disagrees with the first person.

It's easy to read the first persons post either way. That you're lucky they ONLY got 26 corrupt files, or that 26 is extraordinarily unlikely.

I think the "at-rest files through natural causes" gives it away.

Well I've been using NTFS exclusively since 1996 (NT4) and accumulated 870Gb of data.

I've never had a single byte of corruption. Not one, not ever. The data is checksummed about once a month.

26 is a very high number for a deterministic system compared to zero.

My recentish MBP purchase (2011) resulted in a single corrupt file copying 1/3 of that volume onto the machine over the LAN. this was 10.7 at the time. That scared me a little.

The Linux kit I operate has had no discernable data corruption either in the last 12 years.

I know this is an anecdote but over time that's a huge cumulative error.

I can only give you anecdotal evidence, but hopefully it convinces you that hfs+ is the problem.

I've been using Macs for 5 years now. I've stored time machine backups spanning two drives and OS releases from snow leopard to mavericks. (Each drive was used for about 2.5 years, and the OSes were installed soon after they became available.)

Every 9-15 months, I ask Disk Utility to repair the Time Machine volume. The first time, I lost the entire TM directory and had to format the drive and start over. The second time, it crashed Disk Utility because there were so many errors. The third time, I lost the older ~70% of my backups. The most recent time, the operating system refused to even recognize the disk as an HFS+ volume!

Each time, I had the opportunity to format the disk in question and check it for hardware issues. They all passed badblocks and SMART with flying colors. In addition, the volumes were unmounted properly the vast majority of the time (a few instances of human error and power outage)

tldr: HFS+ loses your data catastrophically even if you use it properly.

On the other side of the anecdotal coin, I've been running OSX for a decade now and have in the neighborhood of 100TB of HFS+ formatted media, some in RAID some as single drives. The few times I've catastrophically lost data/drives its been due to the drives physically failing or, in the case of an SSD, a firmware bug. I've lost data to the filesystem (bitrot) but that wouldn't really be prevented by any other FS, they'd just tell me sooner.

By the same token, I've experienced much more corruption on my ZFS (~200TB) and btrfs (58TB) arrays over the last few years than I have on HFS for a decade.

@Alupis -- um HFS+ is journaled, so I'm not really sure what your point is.

You know you're a real tech-y when you get fired up over filesystems!

I've seen similar and noticed drives used for TM tend to fail more than other HFS+ formatted drives. One of my hunches is that SMART is not to be trusted and the repeated spin-up, do stuff, spin down of TM on those drives is the cause. The other is that rarely you will in fact get a garbled data packet over USB where the CRC happens to match.

But when I test the drives with badblocks, nothing shows up. The hardware is fine -- I write bytes and it gives them back to me when I read.

I suspect it's because Time Machine works by creating tons and tons of symlinks, which the HFS+ driver does not like. Regardless, it shouldn't be possible for a userspace application to corrupt the hard drive by creating a lot of links and folders.

Yeah, it's a media problem. There are some missing and interesting pieces of information, was this a laptop? Does it get transferred while off or asleep? What did disk utility tell you about the drive?

HFS+ has had known file system issues and was suppose to be replaced by ZFS, but that crashed and burned. Apple has been working on NFS+ and adding features. HFS+ and NTFS are long in the tooth and really should have been replaced a long time ago.

Can we please stop using lottery/lightning/meteor analogies?

A modern computer does more operations in a single second, than a small country can buy lottery tickets in a lifetime.

Many wildly unlikely things become quite likely with computers.

Reply on news.ycombinator.com