Comment by DiabloD3 | Hacker News Reader

Comment by DiabloD3 | Hacker News Reader

Parent Full thread

Working RAID 5 and 6, RAID "7" (Z3[1], triple parity, where Z/5 is single, and Z2/6 is dual), tiered RAID setups (such as JBOD'ed RAID-Z3s), zvols (as in, but not limited to, swap partitions in ZFS), more nuanced multi-controller and failover/spare setups, write-only journals (ZIL), second tier caching (L2ARC), LZ4 transparent compression, better designed snapshot and snapshot cloning support, more mature CLI tools, support on other OS's (with shared code bases for bonus points), configurable checksum algos, and a few other things that I'm forgetting at the moment.

[1]: Seen that 90 drive Supermicro drive chassis[2]? Three 29-drive RAID-Z3s with 3 hot spares (which is shared across the three RAIDs) in a single storage pool (round robin-esqued), and you could plug that into a 2U with 16 small SSDs (like those Crucial M550 128GBs I use now) for the ZIL/L2ARC farming, and then have, well, near-infinite IO performance. Good luck trying to assemble that with BTRFS.

[2]: A wet dream for CEPH users, too.

Re: [1], isn't that far too many drives in one RAIDZ? Referencing this: http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Prac...

Yes and no. The optimal number for RAIDZ/Z1/Z2, claimed by many benchmarks, is (power of two count of drives) plus (number of drives for parity)[1], with the upper bound being the likelihood of another drive failing before rebuild can complete.

There is a tradeoff of, for example, using 3 RAIDZs each with a third of the drives in your pool, or 1 RAIDZ3 using all the drives in your pool.

In the first case, a rebuild takes less time, but the chances of all data in that RAIDZ being lost is (I think) 4 times higher (but statistics is not my strong point; it may be higher).

In the second case, a rebuild takes longer, the chances of another drive failing before rebuild is complete is higher, however, you require three failures before you have a problem; the array is still fully protected from another drive failure after two drive failures.

The numbers in the example I gave, however, are not quite optimum for performance reasons (you'd be better off with 16 or 24), but most people don't like tons of hotspares (even though that case likely eats drives like old Sun thumpers would).

All of that said, it depends entirely on how much you want data reliability. And, also, it was just an example. Not necessarily a good one. I was just illustrating that ZFS can handle that many drives with ease, but BTRFS would not be a good fit.

[1]: There seems to be no difference between power of two and half way steps in real world performance when the number of drives are big enough, as in, 8, 12, and 16 will all perform similarly.

You're pretty wrong regarding performance here. In ZFS, performance is directly bounded by the number of vdevs given to a pool. If you make one 90 drive vdev, you are more or less limiting yourself to the iops of a single disk. It's a bit more complicated than that, as your throughput increases.

At work where we use ZFS extensively we more or less use # of vdevs * I/O performance of a single drive = total "worst-case" IOPS prior to caching. Caching of course is where ZFS will start to shine - it will win no performance crowns when put up against a "traditional" RAID6 array with the same number of spindles.

This is also why most production implementations of ZFS utilize many sets of mirrors. For example your 90 drive example would be 45 sets of mirrors in our environment, granting roughly 45*120 iops (if NL SAS) total peak capacity. We avoid RAIDZ largely for the reason it would drastically lower the total # of vdevs, and performance falls off a cliff in that situation.

ZFS (not on Linux) seems to handle up to around 300 drives fairly handily before you start to run into issues I've found. 280 is our current max cluster design.

I actually mentioned that, but maybe not as well as I should have. I did say 8, 12, and 16 drives in an array will perform similarly.

However, in cases where you are using SSDs for ZIL/L2ARC, vdev spam isn't as big of a factor. In the example I gave I was illustrating the amount of usable storage you could get in the context of having it reliable, not comparing performance of RAID-Z vs non-ZFS RAID.

Reply on news.ycombinator.com