Your SATA RAID has a 56% chance of data loss
alumnit.ca
alumnit.ca
I'd hate to lose some of the data on that thing, though I'm currently not doing any RAID. Just a disk of important stuff rsnapshotted every 4 hours to another disk.
Also, backing up zfs is really, really easy. 48 lines of shell scripts do daily incrementals and weekly snapshot backups.
But, do yourself a favor and put /usr/gnu/bin first in your path. oh, and alias tar=gtar. Old solarisisms will otherwise be irritating.
If you want something that will actually protect against bit level errors well, generate some extra data using an appropriate Error Correcting Code for your important archives/files. Something like what par2repair generates. This is incredibly slow compared to simple erasure codes like RAID uses, though. There are some filesystems that do this for you automatically, I wrote one:) Actually, other than that I don't know of any filesystems that do this.
If the above is impractical for performance reasons, Google recommends mirroring everything on three disks...
It looks like Joyent was able to rebuild most (all?) of their data in their ZFS data loss situation, but it could have played out drastically differently.
A hard disk has a bit error rate of 10^15:
1,000,000,000,000,000
That's about 16 times higher.
That's horrible.
Taken to a single 750GB disk that means you can read the entire disk an average of 155 times before you'll encounter an error! That's shockingly bad.
I can confirm that the large capacity SATA disks have a noticeably high error rate, which I have never seen before. It used to be that RAID protected you from disk failures, but we aren't seeing that nearly as much now as we are unrecoverable bit errors on otherwise healthy disks.
I have set all our RAID controllers to scrub the data and compare the mirrors every weekend. In the past couple of years we've only had a couple of unrecoverable read errors, but many recoverable read errors.
Recoverable read errors are still bad, the disk gets taken offline and out of the RAID set for up to a minute while the controller retries many times until the data can be read. Then the sector is remapped to another part of the disk.
So yes, it is a real problem, but it isn't too hard to manage in this era of filesystem snapshotting plus high speed RAID6 controllers. In return we've been given huge amounts of storage space.
But yeah, ZFS rocks.
At least you'll know if there was an error.
Are any manufacturers offering a package that contains multiple hard drives but behaves as one?
If performance isn't a factor, is there something I don't understand that makes this impossible?
Instead though, I want it to have a SATA interface so the problem would be solved in hardware. Time Capsule fails if the disk fails (and backups aren't stored elsewhere).
I don't see much difference between SATA and Firewire; in both cases Drobo is hardware RAID.
Time Capsule fails if the disk fails (and backups aren't stored elsewhere).
But then you just make a new backup.
To protect against disk failure, you can use backup or RAID. (RAIDed backup is a little expensive for personal use.) Backup also protects against file deletion or corruption.
* People want increasingly-higher-capacity, smaller, cheaper, thinner, lighter, and cooler laptops. Adding a second magnetic hard drive into a laptop runs counter to all those goals, as does the kind of mirroring described above. The consumer market demands higher capacity over safety.
* Solid state hard drives seem to be the future, at least for laptops. It might be practical to install dual SSDs into a laptop since they are smaller and lighter. It might even be practical to have a rotating set of three as a backup system. Eject hard disk #1, then insert hard disk #3. Hard disk #1 becomes the backup. Hard disk #3 gets synced with hard disk #2 until they mirror each other. Then, relabel the disks 1->2, 2->3, 3->1.
* People are being encouraged to save (or at least back up) their data on the network if it is really important.
My contention is that this is beginning to change. Short of video (and at the risk of being wrong in 5 years) there is nothing to do with a 500GB drive.
I think given the trade-off, consumers would choose safety.
Plus, if you were going to trade 50% of capacity for safety, there are a lot of other kinds of safety precautions to consider. For example, file history versioning (like a local, always-on Mozy or Dropbox or Subversion) would probably protect the user from more data loss than RAID, because most data loss is due to human error, not mechanical failure.
That said, I would really like my laptop to have RAID-1. Actually, I have two 100GB hard drives and an Ultrabay...maybe I will give it a shot on my Linux partition.
It's pretty likely a consumer won't need more than this since that equates to three copies of your data at any given time (laptop, server disk B, and server disk C).
http://www.linuxworld.com/news/2007/061807-zfs-on-linux.html
I think it's 128 bits for truecrypt, although internet searches point to many people using higher (1024 and 4096). And if the bit level error rates are as high as he says, looks bad.