Hacker News Reader

Comment by popknobinfanny | Hacker News Reader

Root on ZFS is an easy sell for me. OpenBSD's ancient filesystem is notoriously flaky, and they have no interest in replacing it anytime soon.

I can't be worried that critical parts of my network won't come back up because the box spontaneously rebooted or the UPS battery ran out (yes it happens — do you load test your batteries — probably not) and their bubblegum-and-string filesystem has corruption and / and /usr won't mount and I gotta visit the console like Sam Jackson in Jurassic Park to fsck the damn thing.

Firewalls are critical infra — by definition they can't be the least reliable device in the network.

View on HN

This is the first I've read that OpenBSD's file system is "notoriously flaky", "bubblegum-and-string" (the opposite of the OpenBSD approach) or make "the least reliable device in the network". The reputation is the opposite.

> visit the console like Sam Jackson in Jurassic Park

Consoles aren't so unusual for most server admins, IME. They're the most common tool.

It seems you’ve read too much general OpenBSD hype material and too little specific information about details like the filesystem. The OpenBSD filesystem notoriously lacks journalling support. It used to support soft updates, but that got removed too. There are no seatbelts. If you suddenly lose power, there is a high likelihood you lose data. OpenBSD is notorious for it.

For those that don't know soft updates are a clever method to prevent filesystem corruption.

Journaling: write the journal, write the filesystem, in event of sudden power outage either the journal will be partially corrupt and discarded or the filesystem will be corrupt and the journal can be replayed to fix it, the problem is that now you are duplicating all metadata writes.

Softupates: reorder the writes in memory so that as the filesystem is written it is always in a consistent state.

So softupdates was a clever system to reduce metadata writes, perhaps too clever, apparently it had to be implemented chained through every layer of the filesystem, nobody but the original author really understood it and everyone was avoiding doing filesystem work for fear of accidentally breaking it. And it may not of even worked, there were definitely workloads where softupdates would hose your data.(I am not exactly sure, But I think it was a ton of small metadata rewrites into a disk full) So when someone wanted to do work on the filesystem but did not want to deal with softupdates, obsd in characteristic fashion said "sure, tear it out" It may come back, I don't know the details, but I doubt it. It sounds like it was a maintenance problem for the team.

Journaling conversely is a sort of inelegant brute force sort of mechanism, but at least it is simple to understand and can be implemented in one layer of the filesystem.

Are you saying softupdates have been removed? When, and what was the name of the file system that used them?

Openbsd's ffs

    Log message:
    Make softdep mounts a no-op

    Softdep is a significant impediment to progressing in the vfs layer
    so we plan to get it out of the way. It is too clever for us to
    continue maintaining as it is."

https://undeadly.org/cgi?action=article;sid=20230706044554

Thanks. So in 2023. I was getting the impression that it was a prior FS a long time ago.

You don't need to talk about other commenters; because you don't know them (with few possible exceptions) you're certain to make a fool of yourself.

The components and their potential benefits aren't really consequential; performance is. Sometimes, the hi-spec components are technologically interesting and exciting to geeks (me too), but have little practical value, especially maximalist components like ZFS. I've never needed it, for example. Very rarely could a journaling file system provide an actual benefit, though I don't object to them.

Sometimes the value is negative because the complexity of those components adds risk. KISS.

> If you suddenly lose power, there is a high likelihood you lose data. OpenBSD is notorious for it.

This can happen on any filesystem unless you have a very good power supply which can buffer flushing of buffers to disk and/or battery backed caches.

Filesystem is one of the rare areas (albeit crucial) where OpenBSD is flaky, tbf.

OpenBSD user here. I wish OpenBSD borrowed WAPBL from NetBSD.

conversely, running a firewall on something like ZFS also sounds like too much. Ideally I'd want a read-only root FS with maybe an /etc and /var managed by an overlay.

Sounds like overcomplicating in the name of simplification. ZFS is a good, reliable, general-purpose system; often the right answer is to just put everything on ZFS and get on with your life.

I’ve had more problems with zfs than all other filesystems combined including FAT. It’s IMO overkill for a root partition.

Problems like? I run zfs on 20gb VMs and a 100tb pool and I’ve never had a problem that wasn’t my own fault. I love root on zfs, you can snapshot your entire OS at a whim. The only other way to get that I know of is btrfs which genuinely does have well known issues.

Not an OP but I have similar experience with ZFS. Over 22 years of maintaining servers, I have had serious issues exclusively with ZFS. My pool is there, but it doesnt want to mount no matter what amount of IRC/reddit/SO/general googling I apply to try and help it boot. After it happened for the second time, I removed ZFS from the list of technologies I want to work with (I still have to, due to Proxmox, but without being fascinated).

As another anecdote:

I've been working with systems for a long time, too. I've screwed things up.

I once somehow decided that using an a.out kernel would be a good match for a Slackware diskset that used elf binaries. (It didn't go well.)

In terms of filesystems: I've had issues with FAT, FAT32, HPFS, NTFS, EXT2, ReiserFS, EXT3, UFS, EXT4, and exFAT. Most of those filesystems are very old now, but some of of these issues have trashed parts of systems beyond comprehension and those issues are part of my background in life whether I like it or not.

I've also had issues with ZFS. I've only been using ZFS in any form at all for about 9 years so far, but in that time I've always able to wrest the system back into order even on the seemingly most-unlikely, least-resilient, garbage-tier hardware -- including after experiencing unlikely problems that I introduced myself by dicking around with stuff in unusual ways.

Can you elaborate upon the two particular unrecoverable issues you experienced?

(And yeah, Google is/was/has been poisoned for a long time as it relates to ZFS. There was a very long streak of people proffering bad mojo about ZFS under an air of presumed authority, and this hasn't been helpful to anyone. The sheer perversity of the popular myths that have popularly surrounded ZFS are profoundly bizarre, and do not help with finding actual solutions to real-world problems.

The timeline is corrupt.)

>The sheer perversity of the popular myths that have popularly surrounded ZFS are profoundly bizarre

Cyberjock sends his regards, I'm sure.

> Over 22 years of maintaining servers, I have had serious issues exclusively with ZFS.

I've been using ZFS since it initially debuted in Solaris 10 6/06 (also: zones and DTrace), before then using it on FreeBSD and Linux, and I've never had issues with it. ¯\_(ツ)_/¯

Not to be deliberately argumentative but still no concrete examples of zfs failures are shown, just hand wavey "I had issues I couldn't google my way out of". I've never heard of a healthy pool not mounting and I've never heard of a pool being unhealthy without a hardware failure of some sort. To the contrary, zfs has perfectly preserved my bytes for over a decade now in the face of shit failing hardware, from memory that throws errors when clocked faster than stock JEDEC speeds to brand new hard drives that just return garbage after reporting successful writes.

> I’ve never had a problem that wasn’t my own fault.

I'm including that. zfs takes more skill to manage properly.

From my understand of ZFS:

When it is treated as just a filesystem, then it works about like any other modern filesystem does.

ZFS features like scrubs aren't necessary. Multiple datasets aren't necessary -- using the one created by default is fine. RAIDZ, mirrors, slog, l2arc: None of that is necessary. Snapshots, transparent compression? Nope, those aren't necessary functions for proper use, either.

There's a lot of features that a person may elect to use, but it is no worse than, say, ext4 or FFS2 is when those features are ignored completely.

(It can be tricky to get Linux booting properly with a ZFS root filesystem. But that difficulty is not shared at all with FreeBSD, wherein ZFS is a native built-in.)

Linux takes more skill to manage than Windows or macOS, yet we all know Linux. zfs is _the_ one true filesystem, and the last one you need to know. Besides that, to know zfs is to have a deeper understanding of what a filesystem is and does.

I will admit though, to truly get zfs you need to change how you think about filesystems.

Interesting, can you share specifics?

> Ideally I'd want a read-only root FS with maybe an /etc and /var managed by an overlay.

OpenZFS 2.2 added support for overlays, so you can have the main pool(s) mounted as read-only:

* https://github.com/openzfs/zfs/releases/tag/zfs-2.2.0

> conversely, running a firewall on something like ZFS also sounds like too much.

this makes no sense. firewalling does not touch the filesystem very much if at all.

what FS is being used is essentially orthogonal to firewalling performances.

if anything, having a copy-on-write filesystem like ZFS on your firewall/router means you have better integrity in case of configuration mistakes and OS upgrade (just rollback the dataset to the previous snapshot!)

my point was that if a hardware vendor were to approach this problem, they'd probably have 2 (prev,next) partitions that they write firmware to, plus separate mounts for config and logs, rather than a kitchen-sink CoW FS

What aspect of ZFS prevents the kind of layout that you envision, do you suppose?

ZFS works just fine with partitions, if that's how a person/company/org wants to use it today.

> Firewalls are critical infra — by definition they can't be the least reliable device in the network.

This is why you have failover for firewalls. The loss of any single device isn't that important.

Sure but you still want them as stable as possible. Needing (emphasis there) to fail over should be for emergencies, not standard operating procedure.

> Needing (emphasis there) to fail over should be for emergencies, not standard operating procedure.

You should be failing testing failover regularly, just like you're testing backups and recovery, and other things that should not "need" to happen but have to actually work when they do.

A good time would be during your monthly/quarterly/(bi)annual/whatever patch cycle (and if there are no patches, then you should just test failover).

That’s why I emphasized “needing”.

Generally it would be part of SOP for updates requiring a service or system restart.

That said, I can't find fault in the filesystem, haven't personally encountered an issue with it, other than it being slow.

OpenBSD's pre-journaling FFS is ancient and creaky but also extremely robust

I am not sure there is a more robust, or simple, filesystem in use today. Most networking devices, including, yes, your UPS, use something like FFS to handle writeable media.

I am not accustomed to defending OpenBSD in any context. It is, in every way, a museum, an exhibition of clever works and past curiosities for the modern visitor.

But this is a deeply weird hill to die on. The "Fast File System," fifty years old and audited to no end, is your greatest fear? It ain't fast, and it's barely a "file system," but its robustness? indubitable. It is very reliably boring and slow. It is the cutting edge circa 2BSD

edit: I am mistaken, the FFS actually dates to 4.1BSD. It is only 44 years old, not 50. Pardon me for my error.

OpenBSD FFS is robust in the sense of 'it is unlikely an FFS bug will lose your data'. But it is ancient, and, well, there are reasons for filesystem developments in the last half century, like 'not losing your data thanks to blackouts or hw bug'.

The lack of at least a journaling FS is inexcusable in a modern OS. Linux and Windows have had it for 25 years by now, and we could argue softupdates are roughly equivalent (FreeBSD has had SU+J for years now too).

Reply on news.ycombinator.com