ZFS High-Availability NAS

> Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought.

I'd recommend this... except that it's really, really hard [0]. And if you think about it, you have to do this for every new application, which is a bit wasteful. So IMO, doing it at filesystem level is a good trade off. It's abstract enough to be useful over a wide range of use-cases (unlike block storage), because if you think about it file systems are just frameworks.

To be fair, block can work really well. I think block storage vendors have themselves to blame. Half the "standards" aren't, interop can be terrible. I know I'd rather have simple, reliable storage - stop stuffing "enterprise" features into a rotting, increasingly technical-debt laden codebase. All back-up solutions for block suck - you'd think this problem would be fixed, especially with cloud storage. I could go on.

[0] https://www.somethingsimilar.com/2013/01/14/notes-on-distrib...

Given the target use case of this kind of system, you control what you can control, which is the infra, because the software will overwhelmingly be industry specific COTS stuff that realistically wont be changed.

Further its likely that the types of internal apps running could handle a recovery period, makes total loss less of an issue...this is "old school" operations - having DR plans, backups and recovery time targets

There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).

This is about a completely different type of VM application, and a completely different kind of storage architecture. This is about everything being on the storage box, not just any application state. The two nodes export NFS shares, which host the root filesystems of the nodes being run on the VM servers. Most likely, the two storage heads will be existing on the same subnet as the VMWare boxes, thus making a network partition that affects both nodes much, much less likely. Additionally, by having the sort of architecture being presented, you can do things like live migrations of VMs, which is useful when the underlying OS needs to be brought down for maintenance.

There are trade-offs with any architectural decisions, and your idea provides a much more byzantine, complex, and expensive system for little to no trade-off at the scale presented. There are quite a few places where they just need a few servers, and going to your idea just makes things more expensive and more complicated.

People will do whatever they want to do. And my comment is not a knock on HAST.

But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain.

Wether it's iSCSI, HAST, NFS, HAST or any combination thereof.

The simplest and least painful? Is to simply create two servers with dual nic's and dual HBA's.

Assign half the disks in a mirror to HBA-1 the other half to HBA-2.

Use one nic for public networking, use the other for a direct P2P Gig-E link between server A and server B.

Use zfs replication over the private network link using the granularity you feel most comfortable with. 30 minutes? 15 minutes? 5? It's up to you.

This is the simplest, and IMO most reliable way to go about this. When you start adding in iSCSI, NFS, HAST etc you are simply making things to complex, introducing multiple points of failure, and the pain when something does break? Ill pass. I'd rather have to manually intervene and fail over to server B if server A fails and vice versa than deal with all the pain of additional things that could break or introduce split brained issues etc.

But it's a free world, use whatever solution fits your comfort zone.

Yeah I think only distributed fs's like Ceph and Gluster really scale. File systems, even ZFS and Btrfs, don't scale because they don't deal with the fact it's not always one or two hard drives dying that'll take out the storage stack. Network card, power supply, motherboard, etc. And unless you have an all SSD array, the recovery times aren't scaling. One dead drive used to take maybe an hour to rebuild and now it can be a day due to capacity growing faster than performance.

> Ceph might be more usable after Bluestore comes along (at the moment erasure coding performance seems to be pretty bad, even with an SSD cache)

I just saturated a 10Gb link in a k=4,m=2 EC configuration on Ceph (Haswell). Too lazy to set up concurrent clients. What do you mean by pretty bad? This is a Hammer cluster without SSD journals or cache.

Bluestore was recently released with Jewel, but shouldn't affect EC performance drastically, if at all.

The amount and type of work that would be required to take a 6disk to a 7disk (in place) is quite hard. If you can take some downtime, you'd really want to snap, send to a new larger system, go down, do the final copy, and come back up. That or work with more vdevs, but then you might have unevenly striped data.

And there are filesystems that are expandable and meet your other requirements, they just aren't cheap. IBM GPFS is one I work with regularly.

You can expand capacity by replacing all the drives with bigger ones. It's obviously expensive and time consuming(has to be done one disk at a time so that each one can be resilvered) but it is possible.

> To add to the downsides, you can't expand RAIDZ vdevs.

> If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev.

This is mostly true, and in many situations for all practical purposes is true. Not fully understanding this recently cost me a few hundred dollars (2x 8TB external drives) when I needed to expand my 4 disk raidz2 to a 6 disk raidz2.

However, what _is_ possible is expanding a pool by incrementally replacing drives with larger capacity equivalents. This wouldn't work in my situation as I already was using the largest consumer drives available - but the next time I need to expand in a few years it may be a possibility.

> ... expandable file system that handles parity/erasure coding as well as bitrot that can be used right now

Unfortunately I believe this to be the case. However in practice, so long as you know up front your data store is not expandable, it's fairly easy to work around without it being too much of a hassle. If having two zpools when you need to expand is a burden for your use case because the data necessarily will be partitioned across pools (ie: in multiple directories), there are tools that will present multiple filesystems as a single mount point to linux.

Alternatively, mirrored vdevs [0] as opposed to a classic raidz2 configuration are much easier to expand, as you only have to replace 2 drives to expand your total storage space (a single vdev) as opposed to having to replace all of your drives with higher capacity drives as with raidz2.

> and Ceph might be more usable

After all that rambling about zfs, the main thing I wanted to touch on: I'd caution most people to not use ceph unless you already know you need it. For the average user, it is introducing so much unnecessary complexity that will just create future headaches when using it in a non-distributed manner (if zfs is an alternative, it must be non-distributed). Which isn't to say ceph isn't useful, it's just built to be optimal at solving a slightly different problem than a filesystem on a single compute can solve.

[0]: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...

Yeah. I have read about zfs to some degree before but somehow missed this until a couple of days ago. For all its flexibility this looks a very bad limitation. I think this and rebuild time (again related to vdev width) makes zfs very easy to mess up.

The goal of bachefs seem to very lofty though there isn't much detail on the implication of raid design. It seemed to suggest it solves the zfs fragmentation problem, gives better r/w performance with a simpler codebase. Anyone who understand the programmers guide well enough to comment on the viability of the architecture? Does it really solves all the problem of a non-distributed huge filesystem capable of working from a single disk to a big array?

I've been looking at SnapRAID and UnRAID which I believe use their own parity and handles expansions. Mostly because I'm currently using BTRFS RAID6 and might move off it.

I also have high hopes for bcachefs. Currently using bcache on my laptop and it's been great. BTRFS doesn't really have an easy SSD cache like ZFS, and it's a hassle to convert over to a Bcache backed BTRFS setup.

REFS on top of Storage Spaces

It means that the vdev on top of the three RAIDZ vdevs can lose up to three devices simultaneously, one per RAIDZ each.

Any further failure would result in a complete data loss. Also, if any of the RAIDZ stripes loses more than one vdev simultaneously, this will lead to a complete data loss as well.

This is key. I mean, I focus on the HP side and used ProLiant servers and storage enclosures in my examples... But the Sun/Oracle enclosures and components are very inexpensive off-lease and come at a price-point where it's totally possible to self-support.

This is not true. Read up on mirrored vdevs. http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...

That is not correct: one can easily increase the size of the pool by adding larger devices. As soon as the last device is replaced, the pool instantaneously possesses the upgraded capacity. No complicated grow commands, no filesystem expansions. It JustWorks(SM).

  zpool set autoreplace=on pool_name
  zpool set autoexpand=on pool_name

properties must be set on the pool for this to work, either before or after the pool upgrade. Note that older versions of zpool(1M), like for example zpool version 15, do not have the autoexpand property.

What cannot be done is a reduction in pool's capacity, but that does not come into these tales.

There is no "apriori size determination", as zpool(1M) uses the entire disks or logical units, and ZFS filesystems use only as much capacity as they require, and no more, leading to extremely efficient disk space utilization.