Hacker News Reader

> How is L2ARC not "true hybrid"?

It doesn't persist across import/export or reboot.

It is demand-filled.

Not all data in the main vdevs are eligible for l2arc.

There is memory overhead for l2arc buffers.

There is CPU overhead in processing l2arc headers.

Once a buffer is in l2arc it stays in l2arc until the underlying data is overwritten or destroyed, or until the l2arc has filled up and the buffer is replaced with fresher data.

A true hybrid in the zfs context would let one pin a dataset or zvol onto a particular vdev, or pin only the (zfs) metadata (or a subset thereof) of a dataset, zvol or pool to a particular vdev.

Openzfs will eventually get both persistence and this form of true hybrid.

Unfortunately automatic migration by zfs of hot data to low-latency vdevs and cool data from low-latency vdevs is not really possible without solving the infamous block-pointer-rewrite problem.

Why should a filesystem care about NVMe?

Because at some point the filesystem becomes a bottleneck. ZFS was designed with the assumption that CPUs would be way faster than storage. When you get speeds over 10GB/sec, [0] you are going to spend a lot of time checksumming all that data.

[0] http://www.seagate.com/ca/en/about-seagate/news/seagate-demo...

Fletcher checksums are very cheap and not a bottleneck.

https://github.com/zfsonlinux/zfs/issues/4789#issuecomment-2...

Maybe I'm reading those benchmarks wrong, but they appear to max out well under 10GB/s. This would mean you'd be CPU bound on your checksums alone with one of those Seagate cards.

That is a Xeon Phi, a modern Xeon with 10+ cores should do tens of GB/s

https://cloud.githubusercontent.com/assets/472018/13333262/f...

https://github.com/zfsonlinux/zfs/pull/4330

Hmm, I for one don't want to tie up ten cores with IO.

If you aren't checksumming your data, is it truely your data?

Who checksums the checksummers? ;)

You can't handle the checksum

Mr Edward Checksum Checker.

Update: At least 1 person didn't get this joke :D

> Hmm, I for one don't want to tie up ten cores with IO.

What else are you going to use the two CPUs in a filer for?

On a single thread.

ZFS is well pipelined for multicore throughput.

Although not recommended, you can turn checksum off. The granularity is at the dataset level, too, so you have a lot of flexibility.

What could ZFS do differently to solve that problem while maintaining data integrity?

Just one idea: offload checksum calculation to a DMA engine. Linux already has a generic DMA engine facility in the kernel, backed by e.g. I/OAT on some Intel hardware.

Assuming this is the same thing as hardware-assisted checksums, both the ZFS on Linux maintainer and Intel have said that they are working on it at various cons last year.

Hybrid storage combines flash and disk for better performance and lower cost. Most hybrid storage is tiered, meaning that data can "live" on either flash or disk, but there are other approaches that look a lot more like a cache (see Nimble Storage, for example). True hybrid storage would work like Apple Fusion Drive or Hybrid Storage Spaces Direct - A single pool with SSD and HDD where data can reside wherever is best.

L2ARC is only ever a cache and although it can be on SSD, ZFS generally won't use much more than a few tens of GB. ZIL isn't even a cache and is only used for synchronous writes. Check any ZFS tuning guide and the gist will be "just buy more RAM or create an all-SSD pool" rather than trying to wedge an SSD into L2ARC or ZIL.

L2ARC and the ZIL can be great in many storage situations, but not all.

We have a TrueNAS appliance with a 480GB L2ARC and a small 120GB ZIL (never fully used) used for image/document storage by our ECM suite. Our metadata usage on the filesystem is astronomical due to having billions of small (<16KB) files, the L2ARC may not do a whole lot for actual data (since most of it isn't hit frequently enough to be eligible to be stored in the ARC/L2ARC) but it's instrumental in maintaining performance with such massive amounts of filesystem metadata.

A write-focussed SLOG device can massively improve your write IOPS though, and is massively cheaper than having an "all-ssd" pool, while being able to approach it's normal use performance. If you have a sustained heavy I/O load, sure - going full SSD will be your best bet, but for most usecases, you only need peak IOPS performance in short bursts.

How reliable are hybrid storage drives? I feel like the last one I saw failed pretty shockingly quickly but not sure (it wasn't mine)...

I'm sorry, I was referring to hybrid storage as a technology category, not the hybrid disk drives like Seagate Momentus XT. You're right that a 2-drive hybrid (Fusion Drive) will mathematically be less reliable than a non-hybrid one and that those "hybrid disks" haven't lived up to the hype.

Pretty much every enterprise storage solution designed today is hybrid (SSD plus HDD) or all-flash and includes lots of advanced availability features.

> "Pretty much every enterprise storage solution designed today is hybrid"

By this do you mean simply that there is both SSD and disk storage around, or do you mean that the storage system transparently chooses where to store particular things without the apps having to care.

Because the latter thing is not my experience.

If you buy an enterprise storage frame there are a variety of mechanisms where as a storage consumer you just see a LUN, but in the backend individual extents/blocks/volumes are tiered in different performing areas.

These systems will transparently "promote" hot blocks to flash or faster spinning disk, etc and demote cold blocks based in usage patterns and policy, without the app being aware.

There are a lot of different ways to do it, ranging from tiering within the array to a storage virtualization solution that can tier data across different storage platforms. In one case I worked on a project where that virtualization tech was used to consolidate 10 data centers to 1, pretty much transparent to the end users and mostly transparent to the folks running apps. (Exceptions were ostly apps that built their own Storage HA)

Ohh okay, gotcha.

I had a ridiculously bad experience with the Seagate Momentus XT drives. Never again..

> How is L2ARC not "true hybrid"?

L2ARC in my understanding is only for reads, whereas ZIL is the write ahead log. Ideally, ZFS would "combine" the two into a MRU "write through cache" such that data is written to the SSD first, then asynchronously written to the disk after (ZIL does this already) but then, when the data is read back, it's read back from the SSD.

This makes sense; if you were building a service that served bytes off of disk, with an in-memory LRU cache, you wouldn't have two separate pools of memory, with one for writes and one for reads.

Did the ZIL and L2ARC concepts come up before SSD was widely available? Especially the ZIL seems very much optimized for crazy enterprise 15k rpm spinning rust. Memory and SSD access characteristics are so different from spinning disks; I don't know why ZFS separates ZIL and L2ARC.

Because the ZIL is a journal log, not a cache. It is intended to increase data security without sacrificing too much performance. Many people also confuse ZIL and SLOG devices though...

By default, the ZIL is written on the same disks as where the data will be stored, but an external device (aka SLOG) can be added. From that point on, your write IOPS will be limited by this SLOG device, so normally you add a more expensive fast disk as SLOG device to increase your write IOPS.

> Did the ZIL and L2ARC concepts come up before SSD was widely available?

Yes. After they realized that their initial claims about not needing such things was bullshit (which some of us had told them at the time) but before SSDs became common.

> I think it's just a package install away on many Linux distros? Also installable on macOS — I had a ZFS USB disk I shared between Mac and FreeBSD.

It's easy to install if you want to use it as an additional filesystem. But if you want to install e.g. RHEL on root ZFS, it's quite an adventure. Even Ubuntu with first-class support for ZFS does not support ZFS on root from the box. Actually I don't understand it. Of all the features, snapshots looks like killer feature for Linux distributions. Make snapshot before upgrade, allow easy rollback if upgrade gone wrong. Something like Windows restore points, but much more reliable.

ZFS root absolutely is supported on Ubuntu and Debian. Every Debian system I've built in the past year or so have been 100% ZFS (around 6 physical systems, but I've also built AWS AMIs this way). You have to install via debootstrap, but it's definitely a working and supported configuration. Installer support would sure be welcome though, it's not trivial!

And that’s exactly the issue. You can’t just use ubiquity to set it up, or even the terminal, you have to go a long, complicated, badly documented path.

If you use proxmox (Debian), you do not need a terminal. And you get all the nice proxmox features. But I agree, standalone setup is no fun.

Actually, I do use the terminal to install it. Boot off a liveusb, switch vty, and go.

I agree, snapshots are amazing for / on a linux machine. Makes backups actually work right, and be very inexpensive (computationally).

But, the tooling is still rather new and untested. At work we are still stamping out upstream bugs that really shouldn't exist, but the ecosystem is certainly getting better. It's only been a couple years since the ZFS on Linux project has gotten remotely any uptake by the distro folks, and even now that interest is rather tepid due to the licensing issues.

If ZFS was GPL I think it would have been the default filesystem on Linux for quite some time now.

> if you want to use it as an additional filesystem

Which is totally fine for a NAS! But yeah, I like my ZFS root on all the FreeBSD installs :)

> snapshots looks like killer feature for Linux distributions

True. FreeBSD and of course Solaris/illumos had boot environments for a long time, it's an excellent feature.

Since Btrfs also has snapshots, it seems perfectly adequate for a root FS. It hasn't seemed important to me to run a ZFS root, even though I have all my user data in ZFS.

btrfs snapshots have a lot of gotchas. Quoting myself:

https://news.ycombinator.com/item?id=14724820

If you have a mandatory flush to disk anytime there's a snapshot, they suddenly become a lot more expensive. They are still atomic in that the operation either did completely happen or not at all happen, if a crash occurs after the command returns, i.e. you don't end up with a partial or corrupt snapshot after you reboot.

The need for a snapshot to be on disk when sending, what you cite says you get an obscure error "stale NFS file handle" not that you get missing files. If you're silently getting missing files on a send/receive, that's a bug and should be reported.

I use snapshots quite a bit both for data, backup/replication with send/receive, and for root fs, and haven't had problems with it. The known problem with Btrfs snapshots is that they are deceptively cheap to create, but become expensive later on to delete due to back reference searching, freeing extents (or not if they're still held by other snapshots) and updating metadata. Dozens to small hundreds aren't normally a problem, in my use case I don't notice performance problems.

Good to know. I've used them only very rarely.

RHEL has a repository run by the ZFS on Linux team, and if you know ZFS it's not much harder than partitioning your disks. The big issue is a lack of documentation on using GRUB with ZFS. It's seemingly nonexistent, leading most guides to say /boot needs a separate partition.

> /boot needs a separate partition

This is true with any Linux-supported FS on most modern PCs since UEFI can only read VFAT partitions.

The ESP does not need to contain /boot, it only needs to contain the EFI. And the UEFI specifications are for an independent filesystem based on FAT, VFAT is still patented. In practice, any FAT filesystem usually works.

You actually don't need a separate /boot - recent grub versions know what to do. You do need a small EF02 partition (I make mine 4096 sectors), and then the rest of the disk as a BF02 partition. I wrote this up for an AWS AMI, but I use the exact same setup on my Debian desktops and servers - https://www.scotte.org/2016/12/ZFS-root-filesystem-on-AWS

/boot and the EFI System Partition aren't necessarily the same thing

> Who has enough money to get a mutli-TB SSD for SOHO?!

https://www.amazon.com/Crucial-MX300-Internal-Solid-State/dp...

I was contemplating a build with 2 of these in a RAID 1 configuration for my next homelab server.

Personally I run a gaming (windows) desktop at home and an always-on UPS-backed homelab server that handles minor ops tasks (mostly backing up side projects and some ETL) + provides dev VMs.

My home office "budget" is ~$2k/year. My gaming/work desktop is ~4 years old, represents ~$2.5k of that budget. Monitors/peripherials/desk/chair generally eats another $2k and are of a similar age. I generally spend ~$2.5k on the dev server. I then usually toss ~$1k into a laptop.

I could easily see someone who purely works from home (rather than 1-2 days a week) operating with a larger budget and genuinely needing a ZFS setup of 2TB SSDs.

Realistically, $2-3k/year is 2-5% of the sort of salaries we see on HN given we make a living at this sort of thing...it isn't surprising people would spend that kind of money to me.

https://hurdlr.com/blog/software-web-developer-tax-deduction...

Keep in mind "business equipment" certainly qualifies for such a dev server so you won't be paying taxes on it if you itemize as well.

> Keep in mind "business equipment" certainly qualifies for such a dev server so you won't be paying taxes on it if you itemize as well.

Even factoring in my homelab spend I don't get any more itemizing than taking the standard deduction as a married individual making $85K, and I spent well close to $2000 on it last year.

> How is L2ARC not "true hybrid"?

I think what that may be referring to is that the ARC is in-RAM and obviously cleared on a reboot, so as a result L2ARC on an SSD is also not persistent. After a reboot, you have to allow the ARC to fill up, then as it evicts data from the L1ARC it's pushed to L2ARC. Until that happens the SSD is not used at all.

ARC is not necessarily cleared on reboot (if doing a "fast" reboot; that is, kernel reload); that's platform and implementation-dependent. Lookup "Persistent L2ARC".

Perhaps "true hybrid" == RAM->SSD->HDD->tape, that is, warm-through-cold storage?

Whatever.

What would be really nice is raw SSD/storage access so that ZFS (or other FS) could manage all the wear leveling and bad block mappings.

Wear leveling is a implementation detail of the medium that can and will change. It doesn't seem right to put that in the filesystem itself. If anything, I would think it should go into some 'generic <specific flash technology here>' device driver.

>. It doesn't seem right to put that in the filesystem itself

It absolutely belongs in the filesystem. When you're doing RAID of any type across the devices, you need that layer to manage the underlying media. A single device view will never appropriately manage wear leveling and garbage collection.

There's a reason companies like NetApp have been working with drive vendors to have more control over the underlying media:

http://www.samsung.com/us/labs/pdfs/2016-08-fms-multi-stream...

It's difficult problem to solve. If we do put it in the filesystem, we need some way for the filesystem to know how the NAND behaves. Otherwise the lowest common denominator dictates and nobody gain anything.

With this in mind, I don't see why the disk cannot handle this in the firmware. As long as there is enough free NAND on the drive, it can manage wear level and GC just fine assuming that it gets TRIM commands.

You've just described why we have storage appliances for high performance and enterprise workloads, and why the drive to standardize always swings back around to customized software and hardware.

It belongs in the filesystem in exactly the same way that volume management belongs in the filesystem.

ZFS includes volume management, _naturally_. I say naturally, but that was a radical idea 15 years ago. Even now it's not universally accepted, but it's quite correct!

I'll add that ZFS is also quite concerned with write patterns IIRC, and thus wear leveling.

The author of the article seems to assume we should all be trusting SSD or "hybrid storage" firmware to properly handle this sort of thing for us like nice black boxes.

I think this was one of the major problems ZFS was designed to solve. To make storage hardware more simple the idea was to move a lot of this logic into the OS (especially as large RAM sizes got cheaper). It's why having a RAID controller sitting under your vdevs is advised against.

Whatever is exactly my reaction :)

You get raw NAND access on like, home routers. OpenWrt/LEDE uses JFFS2 on that.

Change Tape to Cloud and I think you're on to something.

A 2TB SSD costs around 400-500USD. That's not exactly out of the realm of possibility for a small office.

I've got 6x 4TB WD RED in a ZFS RAID10 (3x2 mirrors). I get about 500Mbytes/s read and write, on average. You start getting into 10GbE territory pretty easily with even consumer drives and ZFS. With SSDs, you'd quickly need trunked 10GbE if you want to fully saturate your network - in addition to that consider the client requirements - e.g. each workstation with 10GbE or 1GbE. You'd have to have some pretty decent requirements to necessitate a ZFS RAID10 with [NVMe] SSDs.

The benefit of SSD is not the raw sequential transfer. That is easy to max out with spinning platters. You want SSD for random access, it's significantly better at that.

The ARC (and L2ARC on an SSD) give you the best possible IOPs for read operations given the limits of the underlying hardware. Asynchronous write operations are cached to memory before being written to disk, often sequentially. For synchronous writes, you should use a ZIL on a mirrored SSD.

Sure, but it's a rather high price for just faster storage.

And like… 2 TB of cache is a bit high for SOHO NAS, and if you go full SSD for storage, you'd want two of them for a mirror and that's 1000 USD already…

And we spend what? $200 upgrading from an i3 to an i7 for like... 30% faster CPU performance?

Upgrading $100 hard drive to $400 SSD results in like, 500% improvements in storage speed. If you have any storage-related task... such as video editing, handling of large datasets and whatnot... the SSD will have a far bigger impact on your productivity than any CPU upgrade.

Probably better to spend $150 on SSD cache and $150 on RAM cache than to spend $300 on RAM and $0 on SSD, though. Or the other extreme.

A 2TB drive for 400 or 500 USD? Cheapest ok-ones (samsung EVO) I can find are 700 USD, and then we're not really talking about drives you might actually consider putting in a storage pool used to store critical data.

And if you want 6 or 7 of those puppies, in a non-diy server through a vendor which offers support, 10GE and with enough ECC and CPU to handle that IO load/throughput, you're in for a treat :)

From a protocol level, NVMe is actually designed for SSDs - just look at the amount of queues it has https://en.wikipedia.org/wiki/NVM_Express#cite_ref-ahci-nvme.... To really take advantage of this, you'd need your filesystem to be designed for many independent IO streams. ZFS metaslabs probably help, but I'm sure there is more you can do in this area.

On the other hand, I don't think many people would be hitting any limits where this matters.

Both the L2ARC and the ZIL can be put on separate vdevs (IIRC), to use faster or more reliable SSDs. L2ARC typically wants a pair of striped fast SSD vdevs while the ZIL should be on a mirror vdev of higher-reliability (SLC-like) SSDs.

Also, be sure to have 8+ GiB system RAM available at all times or performance is gonna suck.

> I think it's just a package install away on many Linux distros? Also installable on macOS — I had a ZFS USB disk I shared between Mac and FreeBSD.

Having your root on a filesystem that is provided with your kernel is not ideal situation, update issues make your system basically unbootable.

not* provided?

> How is L2ARC not "true hybrid"?

As far as I'm aware, it's not persistent. Reboot, and your cache of recently accessed files is gone.