Hacker News Reader

Comment by jlgaddis | Hacker News Reader

Ideally, ZFS has exclusive control over the storage. When it's virtualized, it doesn't and there may be various HBAs, raid controllers, etc., in between ZFS and the actual disks. These can (do) "get in the way" and you (can) lose one of the biggest features of ZFS (data integrity and error correction).

View on HN

The keyword is definitely "ideally" :-) With a cloud provider like AWS, storage is always virtualized - so we've always got that working against us. I see ZFS in AWS more about flexibility than data integrity, although having said that ZFS should do just as well or better (certainly no worse) than EXT4, XFS, or BTRFS for reliability. The ability to add storage dynamically without having to move bits around is powerful.

And I've actually had (a couple times now) random hardware failings, with random notification emails from Amazon, on EBS storage on AWS, with accompanying data loss.

Might as well treat your zpool like it's on real hardware and configure raidz accordingly. The cloud does have real, problematic hardware behind it, and it's important to remember that.

[edit] especially if you can configure your block devices such that you know they're sitting on different physical hardware at the cloud data center, you will gain that benefit of ZFS

For a little more detail about this, it's somewhat common for these controllers to lie about when thinks hit the disk. They'll cache it and then write either out but if they lose power or a write gets corrupted zfs will have no way of discovering it until it's too late.

Are other file systems better off with recovery when interacting with virtual disks and host power failure?

They are not, but they are better at swallowing the errors and not bothering you with such details. ZFS fails fast & early, while EXT4 will fail when you realize your Postgres DB is borked.

I guess it's possible that some type of disk command timing could cause unexpected lockups or slowdowns that you wouldn't get with a system that doesn't try to control the hardware to quite the same extent as ZFS, but my (cursory) understanding is that it's rare/hardware specific.

My personal take is that running ZFS on hardware that lies is no worse than running EXT4 on it. YMMV as I'm not a storage expert.

Search online for published papers related to "IRON File Systems." Some researchers injected errors into various parts of common file systems and see how well they recovered. I think ZFS was the best of the bunch though that research is from a few years back and things may have improved elsewhere.

The IRON paper doesn't mention ZFS, I found some citing papers that focus on ZFS but have no comparisons.

Yes you are right. I believe it was a different paper by the same research group... http://pages.cs.wisc.edu/~kadav/zfs/zfsrel.pdf

If the storage lies about syncs, the best you can hope for is replaying a consistent state somewhere in the past. Log structured filesystems with checksums would be a good bet here.

The short answer is no.

Short answer "no" with a longer answer of "ZFS is sometimes better than most other options"

There are virtualization solutions that provide exclusive access to storage. Hyper-V surprisingly is one such solution and I've been running ZFS inside of Hyper-V instances for a couple of years now with no issue.

pass-through device virtualization of various types provide this facility.

Somehow it's a good thing ZFS gets to be tried on AWS, it might trigger some bugs.

"there may be various HBAs, raid controllers, etc., in between ZFS and the actual disks."

Or in the case if EBS - NFS and a network stack. EBS is "Emulated Block Storage" despite what AWS may like you to think.

Source on EBS being NFS?

Does this same idea apply when running ZFS on top of the Linux LUKS mapper (whatever the proper terminology is)?

I haven't tested it with /boot in ZFS, but I have a laptop that has an EXT /boot partition and a LUKS partition with ZFS for everything else (/, /home, etc). It works just fine - and I've never noticed any performance issues with this setup - obviously LUKS has some overhead, but running ZFS on LUKS works as well as EXT4 or BTRFS on LUKS, with all the advantages of ZFS.

there's nothing ZFS specific about this. Your admins should be ensuring that you're not doing concurrent access to shared storage, or if you are, you're using a lock-based cluster manager.

Reply on news.ycombinator.com