The 'hidden' cost of using ZFS for your home NAS
louwrentius.com
louwrentius.com
I've learned the hard way that the 'R' in traditional RAID truly does stand only for "redundant" and not "reliable". Reliability in traditional RAID is predicated an complete, catastrophic failure of a drive such that it is either working wholly and completely or failing wholly and completely.
In a traditional RAID, for any failure mode in which a drive or its controller starts to report bad data before total failure, the bad data is propagated like a virus to the other drives. The corruption returned by a failing drive is lovingly and redundantly replicated to the other drives in the RAID.
This is the advantage of ZFS (or BTRFS). Blocks of data are checksummed and verified and corruption isolated and repaired. Yay for reliable data.
Unfortunately, the three-drive filesystem lasted two weeks before it became unmountable. The only thing that let me mount it was finally running btrfsck. I was left with 57 unrecoverable errors, and lots of lost data.
I would not recommend running BTRFS in RAID5 or RAID6 just yet. Stick with mirroring, if you want to use it, and rebalance to RAID5/6 later on when it's more stable.
I recently got myself a new home server and decided to use FreeBSD and ZFS, so the question is settled for me for the moment. I still hope Btrfs gets there in the not-too-distant future, though.
e: To anyone not up on btrfs, its features are closely tied to the kernel version it's used with. For example, raid56 scrub and device replace, and recovery and rebuild code were not available prior to kernel 3.19.
I also believe the only way to use 5/6 modes before they were stable was to explicitly compile with them enabled. It wasn't just something you could accidentally do.
I didn't have much data to submit. No kernel panics, no useful error messages, nothing beyond it saying it wouldn't mount. One could read the tea leaves from the filesystem as it sat, but such data spelunking could take a while on an 8TB partition, and I wanted to get the disks back into use.
I didn't notice the corruption until after I had unmounted it, so scrubbing it wasn't an option.
As far as I can tell, the only consumer NAS maker that offer BTRFS is Netgear.
I use a HP microstation gen 8, it's lovely hardware; and I use nas4free on a usb stick in the internal port as software.
[0]: by manually I mean the machine that use obnam to backup use one or the other group as a destination, depending on the day.
What I do is have my ZFS in two "layers" (each of them 4 disks in raidz2, i.e. resilient against any two failures), and replace a whole layer at a time. So I started with 4x500GB drives for 1TB of capacity. Then I added 4x1TB drives, total capacity 3TB. Then I replaced the 500GB drives with 2TB drives, total capacity 6TB (and throwing away the 500GB disks, so "losing" 1TB). I'm shortly going to replace the 1TB drives with 4TB drives in the same way.
If you go your own route, look at this server case silverstone cs-ds380b. I think paired with a mini-itx server board with the most sata ports, maybe get some ECC ram, server rated intel processor. I wish I knew the best software route to go. I've considered Amahi in the past and of course FreeNAS. I supect even a couple commercial packages could be sweet to have.
My Setup: OpenSUSE (BTRFS with their Snappy is awesome)
Two drives that are redundant.
CrashPlan for off site backup
I also run my IRC (Weechat) on the box and glowing-bear.org as the front end and this is the BEST thing ever. Run my printer from the server and have a personal R-Studio and Juypter Notebook server running on it. I love it.
If you haven't had a failed disk yet, consider yourself lucky! :-)
This way you get the expandability of RAID with the checksumming and snapshots of ZFS.
I haven't seen it done, it's just a theory, hence why I asked. I'm just not sure if zfs needs to see actual disks, or if it can work on top of any block device, like an md RAID.
MULTIPLE ARRAYS PER TANK
AUTOEXPANDING INTO LARGER DISKS
ALTERNATIVE SOLUTIONS
His issue isn't with ZFS, it's that most parity raid (raidz, raidz2, raid5, raid6, etc) doesn't support safely rebalancing an array to a different number of disks.
With mirrors, the things he describes aren't an issue, especially in a home server. You can start with one disk, mirror it when you're ready; then add additional vdevs of mirrored pairs extending your pool as necessary. Or upgrade two disks to grow a vdev.
http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...
Sun apparently had internally done most(?) of the work on BP rewrite (as it's referred to), but performance sucked rather badly, and the work has not been shipped by Sun/Oracle, let alone anything that came out before the F/OSS code sharing ceased. [1]
(Performance sucked, in particular, for similar reasons to why dedup performance has such a high penalty on ZFS - they end up storing a lookup table that gets hit for every block that gets written after you turn on dedup, and so if that table stops fitting in RAM, it's a bad time - let alone the issue of making sure the disk backing the DDT storage is sufficiently performant...)
[1] - http://www.listbox.com/member/archive/182180/2012/01/sort/th...
Nowadays, RAID5 is deficient.
Each "Home NAS" is going to have very different requirements (to each his own). If you are concerned about reliability and want to be protected against a dual-drive failure, you're better off with RAIDZ2 than with a bunch of striped mirrors, because with RAIDZ2 you can have any two drives fail, where as with an array of mirrors you still have this nagging chance that the second disk in your degraded VDEV will fail and you will loose the entire pool.
Chances are that your home NAS does not have a 4-hour disk replacement SLA.
Also, just blindly recommending to mirror everything may not be the best option for a lot of home users that don't want to loose half of their raw capacity to parity.
RAID-1 a couple of drives, and take backups. If the worst happens, you can restore offline from your backups. If you seriously need 100% uptime at home, in the face of multiple drive failures, you are doing something badly wrong. Or, at least, doing something far beyond 'home use'...
Backups are important, but not ever having to use them is even better.
Not only that, but having RAID or mirror sets gives you flexibility in ways that you might not think about up front. For instance, I just replaced some old 750GB disks with new 5TB disks. I added the new 5TBs to the the mirror set and let it bring all the data across automatically. When it was done, I dropped the 750s out and resized the raid set to use the whole 5TB (I told it it was already clean so it didn't have to sync up a bunch of unused space). Then, finally, I resized the filesystem that was mounted on that mirror set. This was all done live, with data actively being read and written to while all this volume manipulation stuff was happening.
That is why you use RAID at home (and lvm, too).
* Plug new hard drive in
* cp data on to new hard drive
* ...profit?
IMO, much simpler, straightforward, less prone to errors and your data is perfectly safe until you do something with the old hard drive. Again, it's not 100% uptime but this is your home, not a mission-critical business.
And less error prone? Are you joking? In a non-mirror system you have to re-partition and set everything up the way it was, you have to really hope you copy things back correctly (do you have a bunch of hard links? Does cp copy them correctly? What's that option to rsync again? Do you have enough memory to copy hard links correctly? Are xattrs going to copy correctly?). Sigh. On top of that all you can't actually use that disk while any of this is happening. Grrr, and it's going to take 11 hours to copy 5 TB of data back (at 125MB/s)! If you're perfectly happy with your computer being out of commission for 11 hours, be my guest.
Also, when the old disk is removed from the mirror set, it's a complete image copy of the data. I'm not sure why you think that isn't the case...
Seriously, making every disk in your computer a mirror set is really a phenomenal idea. I've been doing it for years now and every time I have to upgrade or replace a disk it's made it heavenly. You shouldn't pooh pooh it until you've actually tried it.
RAID preserves performance and/or uptime in the face of hardware failures. It does not protect against data corruption or deletion. It will happily sync those across all your disks.
Backups protect against any data loss: hardware, software, even intentional. The key feature of a backup is that it is insulated from live data operations (including RAID sync).
In my experience, mirrored and/or striped-mirror type RAID implementations are more reliable, performant, and easier to maintain than any variation of parity RAID. Even for a home setup, it's just not worth my time.
A few years ago I built a home NAS. Back then, maximum drive sizes were 3TB for 3.5", and 1TB for 2.5". The tiny ITX case I had had one 3.5" slot, and one 5.25" slot.
So, I could either do a RAID1 with 2x3TB 3.5" disks; or get a 6x2.5" adapter for the 5.25" bay and do a RAID5/6 with 7x1TB for a net capacity of 5/6 TB.
I'm still not sure whether it was a good idea – the hardware looks a bit adventurous: http://dl.creshal.de/IMG_0873.JPG and was loud as hell –, but it worked well for about five years without any drive failures.
(Then I replaced it with a smaller NAS with 2x2TB drives in RAID, because as it turned out, my interests went from binge-watching storage intensive TV series to reading more books, and I never ended up using more than 1TB storage anyway.)
Many of us here work from home at times, either occasionally or regularly, and we have our own projects in addition to our day job. We each make our own decisions as to what configuration gives us the most acceptable level as performance given the level of risk we're willing to accept.
If a couple of drives in a RAID1 configuration works for you, great! It doesn't fit my needs since my media PC has about 20TB of usable space (~15TB in use) and I don't want to have to hunt through backups of any form when I want to watch or listen to something.
I also have a NAS for my more important files. It's backed up to a cloud provider as well, but the amount of time it would take to download everything after a failure would be very stressful for me, so I spent a bit of money up front for my setup. It's not 100% protection, but so far I've had two drives start to fail, but I was notified in plenty of time to get new drives in and let the array rebuild.
I'm happy with my setup. Well, happy based on the amount of money I was willing to put into it, obviously.
Either way, my setup isn't madness. It's simply different than what you do.
That's business use, it just happens to be at home. Home use is family-style, some documents and pictures, maybe some larger multimedia.
I'm going to hijack this thread slightly to ask: if I want to have a couple of RAID-1 drives running an open source NAS, what's the lowest power hardware solution with acceptable performance?
Depends what you define as acceptable. RAID-1 (as in mdadm) is fairly low on CPU requirements, generally something like an Intel atom chipset (late cedarview chipsets have a TDP around 6.5w) should give reasonable performance, otherwise a mobile/low end i3 would give even better performance.
Other than that, if you're not planning to exploit any of ZFS's functionality, and are looking to deploy on as cheap hardware as possible, mdadm RAID1 (with LVM over the top) is just the ticket. If you need to expand, just add another mirrored pair, assign it a physical volume, add that to the volume group, expand your logical volume, then expand your filesystem (ext4 supports online resize and is very stable, XFS is also a solid contender). All of this is fairly boring compared to ZFS, but it works and it works well..
Since there is a mirrored copy this a correct copy of your data but since there is no metadata or checksums, mdadm has no way to tell which is correct.
ZFS will silently detect the corruption and will rewrite the data restoring the duplicate copy of the data again.
All this works great without ECC RAM, recent studies of data corruption have shown that RAM is much more reliable then previously thought. ECCisnt needed for ZFS for home use.
It's not madness if you're backing up 2-4 home PCs either, storing your movie library. I know somebody that rips their blu-ray disc library to full 32gb video files. That's madness but it doesn't suprise me people do that though. Some of us are hoarders for data. Some people behave like they are the internet archive of the darknets too.
If you've gone to the effort of getting a massive raid like that for home use, you need a backup.
If you don't have a backup, stop reading, you're an arse.
Run a full backup, check restoration.
Blast away your NAS (its ZFS, relies on free space. The fuller it gets the slower it gets (unless you have SSDs or lots of write cache)) It's probably due and upgrade, (OS/XFS)
re-arrange your array, you're changing the way you use it, so you need to do it.
Restore data back over the top (with compression...)
as for "it costs money" well thats why you chose zfs, for redundancy....
The second I elect to host anything even remotly important on it, it'll be backed up, but until then there's no need for me to pay to host that elsewhere.
Professionals who think they don't need backup's because they have redundancy do need a slap though.
What exactly would you rather be using?
Or maybe I need to have large NAS in volume BUT for transient data sets which I could not care less to lose (and thus backup), which makes you an ass for ass-summing.
I'm planning on setting up my first volume pool using 4 5TB Seagate ECs. I can't get any more than four to begin with.
I was intending on using mirrored vdevs, as I'd heard that was basically the easiest way to get started. This is entirely for personal use, and I don't need "enterprise" read/write rates (the hardware I'll be starting with certainly won't manage that), and it won't be the end of the world for (very very rare) resilvers to take all night.
Are there any advantages to not using mirroring, ie using one of the RAIDZ variants? I've already accepted the idea of having to buy new disks two at a time so I can spin up a new vdev with them.
I fully intend to buy disks with widely spaced apart manufacturing dates, so theoretically this should be manageable.
Now I get why 8 disks is recommended... you can make a mirrored pair of 4-disk raidz1s, you get >50% storage efficiency, and you can lose any two disks before it dies.
Never used ECC ram, too expensive. No problem with LSI raid cards, or SAS backplane. Most important advice: People aren't kidding when they say your zpool will slow down when it's almost full.
Online resizing of md arrays isn't reliable, my friend lost terabytes. ZFS is 10 years old, well tested.
Is it best practice that all vdevs in a zpool use the same RAID level?
For example, if you initially created the pool with a single raidz2 vdev, probably the only way it makes sense to add a second raidz1 vdev is if you later changed your mind about how important your data is.
Can we use the absolute cheapest Western Digital Green drives, with limited lifespans? Does ZFS spin down drives when not in use?
Actually, how long do the cheapest drives last when spinning 24x7?
I ask because Best Buy has 5TB Western Digital drives for $110 today: http://www.bestbuy.com/site/wd-my-book-5tb-external-usb-3-0-...
But these consumer drives never list reliability figures, unlike data center drives.
Reds are WD's "NAS" drives, with some advanced features, but the Segate ST2000DM001/ST3000DM001 have worked very well for me with multiple years of almost constant data access.
Sidenote: the upgrade from 4x2TB to 4x3TB was extremely simple, I just failed my drives over one at a time, so as long as you do everything at once you can expand everything really easily with ZFS.
I switched to WD REDs after that, haven't had any issues since.
ZFS is very reliable. Sometimes a scrub would find a problem and fix it. I'd then offline the device causing problems, assuming the sectors were getting "soft", and then I'd run a destructive read/write badblocks on it to flush out bad sectors, then reintroduce it to the pool. The drive would be once again be solid for a good long while.
I'm anxiously waiting for the day inline encryption makes it to FreeBSD's ZFS, as the standard GELI+ZFS method has always seemed a bit clunky to me.
I have 3-4 direct attach usb and FireWire disks. One is a working disk, one backup, the rest for various things.
If one of the working disks fails, I have a local backup. If they all fail, call backblaze.
In all cases, I avoid wasting time and money setting up ersatz infrastructure that in reality is less reliable and more expensive than the simple solution.
However, even that aside, I would likely still use ZFS even with your solution since I make sure of other ZFS features aside just running a software RAID (eg ZFS snapshots, checksumming, send/receive, etc).
My home upload speed is 3mbps.
I make weekly image backups on top of my increment file backups. The images are ~800GB and 200GB. Additionally, I have a ~1TB media collection.
Uploading the images to Backblaze weekly is clearly unfeasible. Restoring from old images in the event of a failure would be extremely time consuming.
Therefore, I have a 14 TB NAS that should last me a long, long while and also serves as a Plex media server. To lose data I really care about requires that four drives / two computers fail or my house burns down. I am willing to accept that risk.
Yeow. This needs a good helping of ZFS snapshots!
In an ideal world, Windows would just use ZFS, but hey, you can't have everything. ;)