Hacker News Reader

I'm currently troubleshooting an issue on my Proxmox server with very slow read speeds from a ZFS volume on an NVMe disk. The disk shows ~7GBps reads outside of ZFS, but ~10MBps in a VM using the ZFS volume.

I've read other reports of this issue. It might be due to fragmentation, or misconfiguration, or who knows, really... The general consensus seems to be that performance degrades after ~80% utilization, and there are no sane defragmentation tools(!).

On my NAS, I've been using ext4 with SnapRAID and mergerfs for years without issues. Being able to use disparate drives and easily expand the array is flexible and cost effective, whereas ZFS makes this very difficult and expensive.

So, thanks, but no thanks. For personal use I'll keep using systems that are not black boxes, are reliable, and performant for anything I'd ever need. What ZFS offers is powerful, but it also has significant downsides that are not worth it to me.

Honestly, pre-made containers are usually black boxes and also a huge waste of resources. If anything, your problem is not using NixOS or Guix, which means you have no reason to waste resources with Proxmox and maintain a massive attack surface thanks to ready-made containers from who knows who, maybe even with their forgotten SSH keys left inside, with dependencies that haven't been updated in ages because whoever made them works in Silicon Valley mode, etc.

You're making a lot of assumptions there.

First of all, I don't see how containers are inherently black boxes or a waste of resources. They're a tool to containerize applications, which can be misused as anything else. If you build your own images, they can certainly be lightweight and transparent. They're based on well known and stable Linux primitives.

Secondly, I'm not using containers at all, but VMs. I build my own images, mainly based on Debian. We can argue whether Linux distros are black boxes, but I would posit that NixOS and Guix are even more so due to their esoteric primitives.

Thirdly, I do use NixOS on several machines, and have been trying to setup a Guix system for years now. I have a love/hate relationship with NixOS because when things go wrong—and they do very frequently—the troubleshooting experience is a nightmare, due to the user hostile error messages and poor/misleading/outdated/nonexistent documentation.

By "black box" I was referring to the black magic that powers ZFS. This is partly due to my own lack of familiarity with it, but whenever I've tried to learn more or troubleshoot an issue like the performance degradation I'm experiencing now, I'm met with confusing viewpoints and documentation. So given this, I'm inclined to use simpler tools that I can reasonably understand which have given me less problems over the years.

Ugh, containers/VMs are black boxes because in common practice you just pull the image as-is without bothering to study what's inside, without checking things like outdated dependencies left behind, some dev's forgotten SSH keys, and so on. There are companies that throw the first image they find from who-knows-who into production just because "it should have what I'm looking for"...

Are they knowable? Yes, but in practice they're unknown.

They waste resources because they duplicate storage, consume extra RAM, and so on to keep n common elements separate, without adding any real security, and with plenty of holes punched here and there to make the whole system/infra work.

This is also a terrible thing in human terms, led to a false sense of security. Using full-stack virtualization increases the overhead on x86 even more with no substantial benefit as well.

ZFS has a codebase that's not easy, sure, but using it is dramatically simple. On GNU/Linux the main problem is not being a first-class citizen due to the license and being a port from another OS, not something truly native even though a lot has been done to integrate it. But `zpool create mypool mirror /dev/... /dev/...` is definitely simple, as is `zfs create mypool/myvol` and so on... Compared to mdadm+luks+{pv,vg,lv}* etc. there's no comparison, it's damn easier and clearer.

I'll bite. I use NixOS as a daily driver and IMO makes the underlying FS type even less important. If my main drive goes I can bootstrap a new one by cloning my repo and running some commands. For my data, I just have some rsyc scripts that sling the bits to various locations.

I suppose if I really wanted to I could put the data on different partitions and disks and use the native fs tools but it's a level of detail that doesn't seem to matter that much relative to what I currently have. I could see thinking about FS details much more for a dedicated storage server

Fs level backups for an OS sounds more relevant when the OS setup is not reproducable and would be a pain to recreate.

Yes and no. ZFS is for managing your data with simplicity and efficiency that isn't possible with other "storage systems" on GNU/Linux. Setting up a desktop with mdraid+LUKS+LVM+the chosen filesystem is a way longer job than creating a pool with the configuration you want and the volumes you want. Managing backups without snapshots that can be sent over a LAN is a major hassle.

Can it be done? Yes. Formally. But it's unlikely that anyone does it at home because between the long setup and maintaining it, there's simply too much work to do. Backing up the OS itself isn't very useful with declarative distros, but sometimes a rebuild fails because for example there's a broken package/derivation at that moment, so having a recent OS ready, a simple volume to send over LAN or pull from USB storage is definitely convenient. It's already happened to me a few times that I had to give up a rebuild for an update because something was broken upstream, few days and that's fixed but without an OS backup, if I'd had to do a restore at that moment, I would have been stuck.

I would certainly feel that way about an ext2 system. But ext4 was released in 2006

There's actually no substantial difference: it's the very concept of a mere filesystem that's obsolete. What's needed to manage your data is:

- Lightweight/instant/accessible and transmittable (at block-level) snapshots, not just logical access

- Integrated management of the underlying hardware, meaning support for various RAID types and dynamic volumes

- Simplicity of management

ZFS offers this, btrfs doesn't (even with LUKS + LVM, nor stratis); it has cumbersome snapshots, not transmittable at the block level, and has embryonic RAID support that's definitely not simple or useful in practice. Ext? Doesn't even have snapshots nor embryonic RAID nor dynamic volumes.

Let me give a simple example: I have a home server and 2 desktops. The home server acts as a backup for the desktops (and more) and is itself backed up primarily locally on cold storage. A deployment like this with NixOS, via org-mode, on a ZFS root is something that can be done at the home level. ZnapZend sends daily snapshots to the home server, which is backed up manually every day simply by physically connecting the cold storage and disconnecting it when it's done (script). That's the foundation.

What happens if I accidentally deleted a file I want back? Well, locally I recover it on the fly by going to $volRoot/.zfs/snapshots/... I can even diff them with Meld if needed. What happens if the single NVMe in the laptop dies?

- I physically change the NVMe on my desk, connect and boot the laptop with a live system on a USB NVMe that boots with sshd active, a known user with authorized keys saved (creating it with NixOS is one config and one command; I update it monthly on the home server, but everything needed is anyway in an org-mode file)

- From there, via ssh from the desktop, with one command (script) I create an empty pool and have mbuffer+zfs recv listening; the server via mbuffer+zfs send, send the latest snapshots of everything (data and OS)

- When it's done, chroot, rebuild the OS to update the bootloader, reboot by disconnecting the USB NVMe, and I'm operational as before

- what if one of mirrored two NVMEs of my desktop die? I change the faulted and simply wait for resilvering.

Human restore time: ~5 minutes. Machine time: ~40 minutes. EVERYTHING is exactly as before the disk failed; I have nothing to do manually. Same for every other machine in my infra. Cost of all this? Maintaining some org-mode notes with the Nix code inside + machine time for automated ISO creation, backup incremental updates etc.

Doing this with mainstream distros or legacy filesystems? Unfeasible. Just the mere logical backup without snapshots or via LVM snaps takes a huge amount of time; backing up the OS becomes unthinkable, and so on. That's the point.

Most people have never built an infra like this; they've spent HOURS working in the shell to build their fragile home infra, when something breaks they spend hours manually fixing it. They think this is normal because they don't know anything else to compare. They think a setup like the one described is beyond home reach, but it's not. That's why classic filesystems, from ext to xfs (which does have snapshots) passing through reiserfs, btrfs, bcachefs and so on, make no sense in 2026 and not even in 2016.

They are software written even in recent times, but born and stuck in a past era.

The scenarios you mentioned are indeed nice use cases of ZFS, but other tools can do this too.

I can make snapshots and recover files with SnapRAID or Kopia. In the case of a laptop system drive failure, I have scripts to quickly setup a new system, and restore data from backups. Sure, the new system won't be a bit-for-bit replica of the old one, and I'll have to manually tinker to get everything back in order, but these scenarios are so uncommon that I'm fine with this taking a bit more time and effort. I'd rather have that over relying on a complex filesystem whose performance degrades over time, and is difficult to work with and understand.

You speak about ZFS as if it's a silver bullet, and everything else is inferior. The reality is that every technical decision has tradeoffs, and the right solution will depend on which tradeoffs make the most sense for any given situation.

How often do you test your OS replication script? I used to do that too, and every time there was always something broken, outdated, or needing modification, often right when I desperately needed a restore because I was about to leave on a business trip and had a flight to catch with a broken laptop disk.

How much time do you spend setting up a desktop and maintaining it with mdraid+LUKS+LVM+your choice of filesystem, replacing a disk and doing the resilvering, or making backups with SnapRAID/Kopia etc? Again, I used to do that. I stopped after finding better solutions, also because I always had issues during restores, maybe small ones, but they were there, and when it's not a test but a real restore, the last thing you want is problems.

Have you actually tested your backup by doing a sudden, unplanned restore without thinking about it for three days before? Do you do it at least once a year to make sure everything works, or do you just hope that since computers rarely fail and restores take a long time, everything will work when you need it? When I did things like you and others I know who still do it, practically no one ever tested their restore, and the recovery script was always one distro major release behind. You had to modify it every few releases when doing a fresh install. In the meantime, it's "hope everything goes well or spend a whole day scrambling to fix things."

Maybe a student is okay with that risk and enjoys fixing things, but generally, it's definitely not best practice and that's why most are on someone else's computer, called the cloud, as protection from their IT choices...

> How often do you test your OS replication script?

Not often. It's mostly outdated, and I spend a lot of time bringing it up to date when I have to rely on it.

BUT I can easily understand what it does, and the tools it uses. In practice I use it rarely, so spending a few hours a year updating it is not a huge problem. I don't have the sense of urgency you describe, and when things do fail, it's an extraordinary event where everything else can wait for me to be productive again. I'm not running a critical business, these are my personal machines. Besides, I have plenty of spare machines I can use while one is out of service.

This is the tradeoff I have decided to make, which works for me. I'm sure that using ZFS and a reproducible system has its benefits, and I'm trying to adopt better practices at my own pace, but all of those have significant drawbacks as well.

> Have you actually tested your backup by doing a sudden, unplanned restore without thinking about it for three days before?

No, but again, I'm not running a critical business. Things can wait. I would argue that even in most corporate environments the obsession over HA comes at the expense of operational complexity, which has a greater negative impact than using boring tools and technology. Few companies need Kubernetes clusters and IaC tools, and even fewer people need ZFS and NixOS for personal use. It would be great if the benefits of these tools were accessible to more people with less drawbacks, but the technology is not there yet. You shouldn't gloss over these issues because they're not issues for you.

Most companies have terrible infrastructure; they're hardly ever examples to follow. But they also have it because there's a certain widespread mentality among those who work there, which originates on the average student's desktop, where they play with Docker instead of understanding what they're using. This is the origin of many modern software problems: the lack of proper IT training in universities.

MIT came up with "The Missing Semester of Your CS Education" to compensate, but it's nothing compared to what's actually needed. It's assumed that students will figure it out on their own, but that almost never happens, at least not in recent decades. It's also assumed that it's something easy to do on your own, that it can be done quickly, which is certainly not the case and I don't think it ever has been. But the teacher who doesn't know is the first to have that bias.

The exceptional event, even if it doesn't require such a rapid response, still reveals a fundamental problem in your setup. So the question should be: why maintain this complex script when you can do less work with something else? NixOS and Guix are tough nuts to crack at first: NixOS because of its language and poor/outdated/not exactly well-done documentation; Guix because its development is centered away from the desktop and it lacks some elements common in modern distros, etc. But once you learn them, there's much less overhead to solve problems and keep everything updated, much less than maintaining custom scripts.

Or you just fully embrace the thin client life and offload everything to the server. pxe boot with remotely mounted filesystems. local hard drives? who needs those?

And the server is handled how? We're always there: complexity can be managed or hidden.

Why do you think some people asked SUN to un-free ZFS back in the day? Because unlike most, they understood its potential. Why do you think PC components today, graphics cards first, then RAM, and NVMe drives after that, cost so much? Because those who understand realize that today, a GNU/Linux homeserver and desktop are ready for the masses, and it's only a matter of time before a umbrel.com, start9.com, or even frigghome.ai succeeds and sweeps away an increasingly banning and therefore unreliable and expensive cloud providers. Most still haven't grasped this, but those who live above the masses have.

Why are snaps, flatpaks, docker etc are pushed so hard even though they have insane attack surfaces, minimal control over your own infrastructure, and are a huge waste of resources? Because they allow selling support to people who don't know. With NixOS or Guix, you only sell a text config. It's not the same business model, and after a while, with an LLM, people learn to do it themselves.

Most people will use LVM+Ext4 in the enterprise for backups and easy resizing and the like. Also Guix has rollbacks for the whole package manager and system so whole backups aren't mandatory, the data would be handled and backed up externally with some LVM volume, for sure there's a Guix service for backups.

Well, yeah, but in the enterprise world, the infrastructure is often really pathetic, layers of legacy garbage with "do not touch, if it breaks everything is lost" labels. I wouldn't take them as a model, which is one of the common and self‑harming behaviors many people practice: trying to imitate others just because they're big, not because it makes technical sense.

The system can be regenerated, sure, but it might fail to rebuild at some point because of a broken upstream package; that's what backups are for, and it's handy to have them for everything. Adding, say, thirty gigabytes for the system costs little.

LVM does allow you to operate, yes, but with much heavier limitations and far more inconvenience than ZFS... And here we come back to operations vs. even top developers, like Andrew Morton's infamous quote about ZFS, which shows the difference between those who genuinely manage complex systems and those who only see their own workstation and don't understand broader needs to the point of denying they exist even for themselves.