In OpenZFS and Btrfs, everyone was just guessing
phoronix.com
phoronix.com
With the exception of a couple people like Jim Salters, I don't take any advice I see about ZFS unless I know the person is a dev or has commits on the project. And most of the time I take the advice I've seen 'ryao' post which is, paraphrased, "use the defaults unless you know what you're tuning". That should go without saying, but there are a lot of armchair "geniuses" posting online about ZFS and they are just guessing.
And, how to you get any new developers if the expectation is an immediate, complete understanding of the system? That just isn't realistic and, in most cases, the dev that knows a little bit probably knows exponentially more than an average user. Interacting on the bug tracker and debugging things they don't completely understand is going to improve their understanding. There's value there.
Things have gotten more complicated too. We aren't using spinning disks without caches any more, right? I can understand a disk from 20 years ago. They're relatively simple. I couldn't tell you the first thing about SSDs or NVMe stuff.
There are going to be bugs. Just be glad the devs acknowledge them and fix them. Personally I still rank ZFS as the least likely to lose my data and I'll keep using it. I put roughly zero value in a forum full of people complaining.
Here, have at it: https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...
So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference.
It's a shame a bug crept in to an otherwise pretty much faultless filesystem. I'm sure there are some truths in there about commercial support or not running the bleeding edge or whatever. And making backups, even for ZFS backed filesystems. Sacrilege, I know.
Unless I'm just parroting truisms here, dedupe is almost never a good idea and compression is a much more common case-by-case.
> So personally I'd encourage messing around (on something non-prod, of course). Measure measure measure! Does it actually make any difference.
100%. So much depends on the particular workload and resources.
> And making backups, even for ZFS backed filesystems. Sacrilege, I know.
Logical backups, specifically. We have seen more than once corruption and data-loss in replicated encrypted snapshots due to bugs in ZFS.
It is cheaper to buy more disk than the memory to enable dedup (somebody could run the numbers). ZFS ARC also likes lots of memory and you get good performance gains with more memory allocated to it.
In other words, turn on LZ4 compression everywhere and don’t think about it otherwise.
Dedupe can be used to infer secret data. .. more news at 11
My personal approach to filesystem evaluation is pretty much "How long has it been since I've seen someone talk about a data loss bug". This is my first time hearing one for ZFS. BTRFS seems to have at least one post a year. Ext4 has almost none, but maybe that's because people trust it and blame hardware when unsure what the issue is.
This [0] is a brief post on MySQL, but most of it applies equally to Postgres.
[0]: https://www.percona.com/blog/mysql-zfs-performance-update/
https://openzfs.github.io/openzfs-docs/Performance%20and%20T...
compression=lz4 recordsize=32K
In postgres -> full_page_writes=off
atime is the most useless filesystem feature I’ve heard of. But on most Unix-like systems it’s enabled by default and the main thing it does is turn reads into writes, degrading performance. All the while ctime isn’t a standard Unix filesystem feature. And ctime to me always seemed infinitely more useful (and cheaper!) than atime.
</rant>
Finding old objects in a cache is an issue specific to caches and should be implemented there.
Mind, you shouldn't really change the record size without knowing your use-case. Make a PostgreSQL file system, set it's recordsize to 8k. That's a good idea.
Not so much a good idea doing that to all your filesystems. Stick to the defaults without a good justification to deviate.
Reddit, random blogs and official documentation (often that by Oracle) turned out to be much better resources.
If you want to use ZFS, then TrueNAS with sensible defaults is a great way to go. If you want to learn ZFS enough to roll your own deployment, change the configuration correctly and understand the changes you make then don't start with TrueNAS, and definitely don't listen to the TrueNAS community.
Or at least, that's how I read the GP comment.
I know, I was doing just that 5+ years ago.
Commercial storage vendors aren't really any different. Mostly the biggest difference here is they offer a very limited subset of hardware and disk configurations and tend to iterate over that very small configuration space. Step outside of that config space and they won't have much insight.
This remind me of the earlier days of Mac vs PC. Mac tended to have a very small hardware set of supported devices, and in many ways helped stability. PC supported pretty much anything you threw at it, which commonly had issues with crappy drivers and untested interactions.
I evaluated TrueNAS Scale about 6 months ago and bailed on it very quickly. My install with their "sensible defaults" ended up having swap partitions on all my spinning disks. That alone was enough for me to decide against using it. One bad choice likely means other bad choices in my experience. They also trampled / reset my 'zfs_zrc_max' tunable every time a VM got started (I think) and it wasn't obvious what was doing it.
I didn't think there was any value in learning the quirks of TrueNAS over doing a setup from scratch, so I ended up going with Ubuntu LTS and doing all my own config / testing.
Eventually, someone explains: “just try it, or stop complaining.”
30s later, they exclaim: “holy ** I’m getting 60 fps now!”
Sometimes, what looks like misinformation isn’t misinformation.
TL;DR: this game switches to a software rendering pipeline on “medium” or lower. You have to set it on “high” to switch to the hardware rendering pipeline.
Commenter posts a nice summary of how ZFS will try a quick compression algorithm to see if a block is even compressible and then try other more expensive ones to achieve better compression.
Another commenter asks how they know that.
First commenter replies that they wrote the code for it.
Nothing more than a ZFS user (to some extent my company relies on it even) here, so I just try and color inside the lines, but I'm ever impressed by it.
No, that is not plausible. It's a filesystem. Behaviour in all cases for a filesystem should be predictable. There are zero things a user should be able to do that would make a filesystem non deterministic.
Saying the zfs/btrfs teams have no idea means the person has no idea about software development process ... It is a system none fully knows a system that complex.
If they did, the bug wouldn't have happened.
When filesystem or databases have serious bugs, they are often heisenbugs. Incredibly hard to pin down. You need to be able to replicate the bug to find what is happening.
In one of the first jobs I had, large Oracle database started to corrupt data repeatedly about once a month or so. Oracle had to send on-site team to monitor the database operations to catch the bug in action. They basically had their own datacenter to record all the logs from database, networks and operating system.
It took weeks for them to catch the bug. It was related to certain network driver in the operating system and the bug manifested only with some specific network traffic pattern.
But the issue was, this was happening even after we removed all instructions to delete data from the flash.
The device had most traces between controller and flash completely hidden as a precaution for hacking/snooping, making it extremely difficult to diagnose the issue as the issue completely vanished in a test harness when the flash was connected to controller outside the board.
The problem also happened exceedingly rarely -- we needed about 100 of these devices to run constantly a certain operation and it took us to wait for about a week until one of these devices died on average. But we had millions of them in the wild and we would get hundreds or thousands die on our customers every week.
It took half a year of debugging and tens of millions of dollars in losses.
We found that one of unrelated components on the board emitted too much noises. And a software change to speed up writes to the flash removed a special sequence of that preceded and followed every one word write to the flash. This special sequence protected the flash from accepting a random noise as a command. Usually, the command would be something harmful or meaningless. But sometimes it would be interpreted as a real command -- and one of these was a command to clear entire flash chip.
I heard from a coworker that an older version of the product had bumped from a 16MHz chip to 20MHz to run more code. Chip supplier screened parts for the higher speed, but didn't cover higher speed across the normal voltage range. When supply dipped (within the old spec) switching speeds slowed. The first circuit to fail was something in the branch condition logic. This caused all conditionals to not branch, which lead to running all code in the memory space in order. That included a few instructions to.... fire the airbag. It was a rare thing, but a very big deal.
“Business woman on plane: Are there a lot of these kinds of accidents?
Narrator: You wouldn't believe.
Business woman on plane: Which car company do you work for?
Narrator: A major one.”
How do you even protect against such a thing you just described? Put the airbag on a diffent microprocessor?
Of course it's possible to create more and more improbable scenarios. That's not to say those never happen; I've seen really improbable occurrences.
(We may have learned a bit about noise and improved further iterations of the device, but sw change was enough.)
We couldn’t change the data on it. Linux would report that data was written to it, but when we unmounted and remounted it, the data was unchanged.
2. Noise can definitely affect other components. Traces are antennas that both broadcast as well as receive signals.
As technology marches forward, the voltages and tolerances for communication become smaller and tighter. At the same time, you integrate more functionality on the board so maybe your sensitive MCU and flash and ram are all living close by to some radio equipment and PSU and whatever else. Power interruptions to certain components can create momentary flashes of noise that can act as EMP bomb going off. On other occasions a noisy, badly designed high frequency power component can emit noise that is picked up by some random traces due to their geometric configuration and length.
Your comment suggests you never worked on a large project with thousands of components and dozens of major chips?
That's one failure per 16,800 hours, or 700 days. That's pretty goddamn rare.
Fortunately, we already had all the design, development and production resources to do the project on site (yes, we were doing it all, not outsourcing to China).
Second, a story from many years ago.
The local ACM chapter I was a member of bought a whole bunch of e1000 NICs, back in the day when 1Gb at relatively affordable prices was new, and a switch, and threw them in our servers and desktops, which were a motley assortment of Intel, AMD, and Other things, running various flavors of Linux and Solaris.
_Some_ of the Linux systems had a problem where eventually, the NIC would stop responding and dmesg would flood with "Tx Unit Hang" or "Rx Unit Hang", and not work again until a reboot.
If we swapped the NIC from a system where this never happened to one with the same model NIC where this happened, the problem kept happening - that is, it was something about the system+NIC, not just the NIC.
Intel eventually gave us a shipping label to borrow one of our systems when they said they hadn't been able to reproduce it but kept having people report it, and we said "we have a number of them".
They made some really exciting noises after some debugging, and eventually concluded the issue was that our really cheapass Sempron board that we sent them screwed up badly if you tried negotiating 64-bit PCI in their normal PCI slots, and would sometimes miss the messages between the NIC and the host saying that Tx Ring buffer 0 is free/full, etc, and eventually, it would miss so many of those that it would think the NIC had no more buffers, and here we are. They couldn't reproduce it because all their testbeds were well-designed and tested things.
Choice quotes mid-debugging include "I do not see how anyone could make a PCI device work reliably in this system".
They did figure out a workaround, and ship it, and asked if they could keep the system permanently for future testing.
But that was exciting.
In a previous life, I was paid to build storage systems for some HPC-ish workloads.
So I was testing a bunch of cheap-ish desktop drives that had really nice (for the time) sequential throughput, supposedly, and threw together a Supermicro system with a Xeon, ECC RAM and some SAS HBAs and several external enclosures full of these disks, made a couple of raidz3s, and started trying to stress it.
I quickly found that rarely but somewhat reliably, I'd get correctable checksum errors, and because it was raidz3, it always had a lot of spare recovery bits, but the numbers kept going up...and went up even after a scrub finished and "corrected" all of them.
Well, SAS has checksums over the wire, so I'd be seeing disk errors if the wires were eating my bits, and it was across all the disk controllers, so either they were all bad or it wasn't the controllers, ECC RAM and no correctable or uncorrectable events fired...
This predated ZoL, so this was originally on early illumos - I then tried FreeBSD, and it did the same thing.
Huh.
So, the drives in question were Samsung HD204UIs, which, it turns out, have a really spicy firmware bug, where if you send them a SMART IDENTIFY request with data in the write cache (e.g. it already told the OS it was stably written out), it just...dropped it.
So the background smartd I had running for collecting data on the disks was causing them to eat some of the writes whenever that lined up.
The funniest part was, though, that Samsung released a firmware update, but it doesn't change the reported firmware revision, so you can only know by testing if the disk is going to eat your data like that.
I believe smartctl still prints a big warning to this day about all this if you ask it about those drive models.
So, I have a little old SPARC64 niagara 2 box, which is a strange beast for a number of reasons.
The onboard NIC is a strange bespoke Sun-created Ethernet card, over the PCIe bus.
Since it's a SPARC in 202x, it's not really well tested or supported, so updating is always fun.
One day, I lost power, and on next boot, it booted a newer kernel, and the kernel panicked.
I rapidly determined this was from trying and failing to initialize the NIC, and attempting to bisect it turned out to be really exciting - after getting impossible bisect results, I realized, once I booted a "working" kernel, all successive kernels worked until a cold boot.
The problem, you see, was that they had changed some things about the PCIe initialization to enable certain kinds of memory protection features.
The actual memory protection feature works fine on this hardware - but the _check_ of whether the feature is supported causes the machine to panic.
But once the machine is booted, it doesn't try to re-initialize this on warm boot, so it doesn't trip this problem.
Found a similar issue in a switch stack relating to multicast stream delivery. Knowing about this class of bug and having isolated it down to the switch, as the only possible device in the path, being responsible, I filed a ticket with the appropriate team to get the stack rebooted.
They were understandably skeptical of my supposed prescience and so I had to spend a considerably larger amount of time writing test cases and running deep packet captures in order to prove that a single bit, in the port field, was getting flipped under circumstances that were repeatable but not entirely explainable without even more work.
Finally, they relented and agreed that, just turning it off, then on again, was indeed the best initial solution. Fortunately with all the test harnesses in place we were able to prove that it cleared the issue on that set of ports.
I'd go as far to claim this: If you're shipping a product that relies on networking at your customer's premises you basically need to maintain a list of cards and drivers that you support. Especially if you're doing anything weird, where "weird" means anything but bog standard TCP.
Additionally, unless your system has an iommu and configures it to strictly limit your NIC, most NICs can DMA to any address in memory.
Unintentionally providing the wrong address to receive a packet with is a great way to corrupt a database.
I'm sure there's all kinds of nuance to your tale, but it still sounds like Oracle's bug to me. They just found what it was in your system that introduced the behaviour they hadn't anticipated, and then got you to remove it.
It looks like they managed to exactly pin down what was happening.
> Signed-off-by: Rob Norris <rob.norris@klarasystems.com>
> Sponsored-by: Klara, Inc.
> Sponsored-by: Wasabi Technology, Inc.
"Sponsored by:" search in FreeBSD commit messages: https://freshbsd.org/?q=%22Sponsored+by%3A%22
It also helps with lawsuits, which was a major thing right at the beginning of history of BSD.
* https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc.....
Linux later had the lawsuit issue with SCO and IBM and tracking where certain things came from (which was not helped by the fact that Linus Torvalds refused to use source code tracking for the longest time (later taking up Bitkeeper, and later developing git)).
You likely mean the end of the effort to replace all AT&T code, which was more than a decade later.
Then ZFS has one bug and everyone starts acting like the sky is falling. And sure, it was a bad bug, but you had to be pretty unlucky to trigger it, and it was present in ZFS for all of a month.
Meanwhile, people's Google Drives are apparently forgetting the last the eight months of changes.
OpenZFS is fine.
I suppose it's still a bad bug, but it goes to show that file systems are complicated beasts, practically impossible to test all the ways they can be put through the wringer, and bugs like this can hide for 17 years before being unearthed.
Looks like you might be right about that. The oldest commit referenced in the fix [0] was from 2006[1], which was ~5 months after Sun released ZFS.
Glad this is fixed!
[0] https://github.com/openzfs/zfs/pull/15571
[1] https://github.com/illumos/illumos-gate/commit/c543ec060d
Drawbacks and tradeoffs abound, but outright “bugs” doesn’t seem like something ZFS is generally associated with.
After some initial bumps in brand new software, the Sun kernel group did a good job of avoiding corruption. Once Sun was absorbed by Oracle, OSS ZFS moved into illumos, who were overall quite good at doing the same (although they had less resources to play with). OpenZFS brought ZFS to the rest of the world (good), but I can't help but notice that the illumos devs are increasingly worried about pulling changes back from OpenZFS into illumos.
The increased popularity is for the best, but the change rate has increased, with all that entails.
And i think that OP is right, development speed is really a bit too fast atm for a filestystem. Maybe a codereview ala openbsd could do the job?
Maybe sometimes once in 2-3 decades it's worth taking the lessons learned and re-doing the thing? Simplifying and streamlining it greatly in the process?
Like, this bug, the file holes - do we even need them now, with very good compression for once?
ZFS has literally all of these things. I’m not sure if he’s just new to the community or making really poor assumptions based on his experience with btrfs.
Heck just look at GitHub. Anytime a new feature is being developed part of the merge process includes results from the test suites…
Ok. So, We hit the deadlock that was fixed here:-> https://github.com/rohan-puri/zfs/commit/8e4d086c13c16bc461b... but it never got merged to the master or release as far as I see it Current Master: https://github.com/openzfs/zfs/blob/acb33ee1c169bf1c1f687db1...
When I look up the problem, I could only see the issue being discussed, and probably leading to that commit, and then it was not in my current release (2.1) despite several years later. I’m wondering if ZFS still holds that high standard for reliability.
https://github.com/openzfs/zfs/commit/fd7265c646f40e364396af...
git branch --contains fd7265c
* master
git branch -r --contains fd7265c | head
origin/HEAD -> origin/master
origin/compat-5.15-META
origin/issue-14573-backport
origin/master
origin/revert-14721-remove-quota-zap
origin/zfs-0.8-release
origin/zfs-0.8.7-staging
origin/zfs-2.0-release
origin/zfs-2.0.3-staging
origin/zfs-2.0.4-staginghttps://bugs.freebsd.org/bugzilla/show_bug.cgi?id=275308
Second, I don't like the editorialization of this title ("In OpenZFS and Btrfs, everyone was just guessing") at all. No, nobody was "just guessing", but as far as I know there is no featureful FS that has undergone formal verification. It's a large codebase started by a now long defunct company that solved critical problems and delivered a lot of value, but there absolutely can be issues lurking over long time periods. It's a testament if anything that work and usage did uncover the issue, it wasn't brushed away at all but instead drilled down on and solved in short order, and now the already extensive test suites are expanded again in an organized way. And spekaing of "critical problems":
Third, a lot of the commentary around these sorts of things seems to indulge in noticing the rare misses while ignoring many hits. Amongst other reasons part of the core motivation for me to switch to ZFS 100% in 2010/2011 or so and doggedly stick with it ever since was precisely because I experienced data rot (permanent corruption) with my data under previous filesystems where that WAS NOT A BUG. HFS/UFS/NTFS/XFS/whatever, none of them offer any guarantees of data integrity in the first place! Bit flips somewhere, hardware has issues, copying has noise, whatever? RAID-5 write hole? Those or lots of other things are not a bugs at all in old filesystems, because that's just how those primitive things were. I've been carrying forward data since my first Apple IIe, and went back to find some of my old work, early digital photos and drawings I cared about, and somewhere along the line it had gotten mucked up. I know not where or when, because there was no chain of checksumming and trust that would have a chance to alert me. It's impossible for a human to keep up with terabytes of data manually, it has to be fully automated, baked in. At least in ZFS there being any corruption is a drop-everything big deal problem that is incredibly rare and niche and serious people will care very much about. Pretending the old stuff was good or even acceptable is pure bullshit.
It doesn't have to be mathematically perfect to deliver value, and importantly to be far superior to everything that came before. And it's inarguable that it's been battle tested very very hard for a very long time at this point. It's certainly saved my bacon a few times.
The whole weird implication that there's something else which is perfect is always bizarre: there isn't, everyone knows that, it's extremely non-trivial to do that.
Got bitten by that one once as well. I recall the XFS FAQ had something like in it like: "If you know it hurts, then don't do it."
I remember that happening to me once long ago, but never on rhel 7 or above.
- https://access.redhat.com/solutions/197643
Also, there's a PDF doc (from a conference presentation), covering many layers of potential alternatives for ZFS features:
- https://news.ycombinator.com/item?id=38484598
But "just" slapping all these layers on each other (configuring each individually and then hoping for the best) doesn't seem like the best idea for production systems either.
I guess back in the Sun/Solaris days, people still had the luxury of releasing (major) features in major versions, every few years instead of weeks or months... and as far as storage, backup, security solutions go - can't really do this right with the move fast & break things culture...
I'm going to spend a moment praising scripts though...
A dozen or more years ago I was running a bunch of backup servers. The backup system was rsync to a zfs, then snapshot the zfs. Then do a rolling delete of snapshots to achieve X daily, Y weekly, and Z monthly copies. This is because the hardlink trick is amazingly inefficient.
I started off using Nevada, but had issues and no real expertise there, so I switched to Linuz with ZFS fuse. That worked about as well as Nevada, but about once a month or two zfs-fuse would die.
I built a script that would simulate the access patterns of the backup systems. I then went through a few iterations with the developers of running the script (which logged its actions as shell commands, so the log could be re-executed), reporting the results to the developers, and them producing a fix. After a few back-and-forths with the developers, zfs-fuse became rock solid for my use case. And the script was the key.
https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle
ZFS has always appeared to me like a bloated mess. Btrfs is more minimal, but too much so, in that it doesn't do parity (at least very reliably), which sucks because I don't always want mirroring. And it has a reputation for gobbling your data up.
If bcachefs can provide all the most useful features of ZFS, while being minimal and baked into Linux by default, man that will be amazing.
I think it's probably worse than usual with ZFS, mostly for two reasons: a lot of hype (justified or not) early on its life, leading to hype aversion and/or backlash, and a "sour grapes" effect because its license ensures it will never (or at least until its copyright expires) be merged into mainline Linux, so those who can't or won't use it (due to a need or desire to run only mainline Linux kernel code) feel left out and/or get annoyed at seeing it pushed as the best thing since sliced bread all the time.
Adding to that, its main competitor (btrfs) is known for having had (real or perceived) reliability issues early on its life, making its use more controversial (and its proponents more defensive at seeing the same reliability issues being raised all the time).
And file systems are important, since people are reasonably concerned about the integrity of their data. They are also not easy to migrate (other than a rarely used tool to migrate in place from the ext2/3/4 family to btrfs, migrating filesystems usually requires dumping and reloading the whole data), making the choice of filesystems an important decision.
If you trust a single filesystem with your data, you're wrong. It doesn't matter how fancy it is, how many parity blocks it has, or how many disks it's backed by: a single logical filesystem definitionally cannot be redundant. If you pretend it is, it will blow up in your face one day, and it will be nobody's fault but yours.
My backup is a drawer full of individual hard drives. Unlike any filesystem that ever has been or ever will be invented, that drawer cannot corrupt my data.
Even if user's should use multiple redundant filesystems (lol ok), that doesn't invalidate the criticisms in the linked forum post.
Yes, absolutely 100% at fault. There are many failure modes here beyond corruption: human error, electrical surges, flooding, fire. Hell, somebody could break into your house and steal the NAS!
Filesystem bugs aren't a big deal. If they are a big deal to you, it's because you're trusting a system with a single point of failure, and the consequences of that inevitable failure are not acceptable to you. Stop doing that.
If I could choose between introducing a random filesystem bug and a random wifi driver bug into the kernel on my laptop, I'd take the filesystem bug every day of the week and twice on Sunday. Reinstalling my laptop is at worst a minor inconvenience that takes me an hour, and I can always use a different filesystem until the bug is fixed.
It's your fault for not having 2 extra phones on you in case your phone bricks itself at a bad time.
It's your fault for not encrypting your signal messages by hand, encryption bugs aren't a big deal, if they're a big deal to you, it's because you're trusting a system with a single point of failure, and the consequences of that inevitable failure are not acceptable to you. Stop doing that.
Your position sounds insane to me of applied to any other piece of technology
You also missed the most important part of what I said:
>> because you're trusting a system with a single point of failure, and the consequences of that inevitable failure are not acceptable to you.
I don't care if my phone fails: it's disposable. Sure, there are times it might be inconvenient, but it's not like I need it to drive my car or get into my house.
If your phone failing is some huge big horrible problem for you, yeah, you absolutely ought to carry two phones around. It's unreasonable to expect consumer grade technology to be that reliable.
Backups failing mean 15 years of family photo memories forever lost.
Risk mitigations should reasonably scale in proportion to the cost of loss.
This is a very easy problem to solve: don't do incremental backups. Or have N backups and rotate, which isn't as good but still gives you more time to notice. Hard drives are cheap.
Usually you have to dig through the git blame until you discover the commit message is nothing but “Merge fix-something into main”, from there you have to crawl the commits to find a PR number. Inside the PR, the only useful data is “Fixes: #1234”. From there, if you are lucky, you can find some rationale hidden among the 100 comments from stale-bots. If you are not lucky, the info you need is described in another issue of another library that is cross-linking to this issue.
For something sensitive as a file system, hearing they are no better is extra scary.
Store started complaining about sometimes scanning muffins - this was where the price was embedded inside the barcode - the prices were ridiculous for muffins but the barcode has a luhn check digit.
Must be our code management said - so we looked at our serial driver until we eventually found evidence through logging all serial I/O traffic that it was the barcode scanner that was sending back a corrupted barcode.
I would more worried what you can't see - SSD controllers - yikes don't want to think about the bugs in those things.
I don't believe that "everyone is just guessing". There are some pretty knowledgeable folk that work on this.
[1] I've triggered some corruption in snapshots on an encrypted pool. No permanent problems resulted and no data was lost.
[2] I provided a very minor documentation fix that was encouraged and promptly merged. https://github.com/openzfs/openzfs-docs/pull/472
OK, one of those is not true but this is how ALL software of any complexity is. I'm also old and ornery like the OP and I don't like it and if I get drunk I will talk hella shit about it but it's just a fact.
Software sucks. There was indeed a time when we (collectively) did a lot more to try to prove it worked before shipping it. But then:
a.) the money people realized it made money if it mostly worked, and made less money if we went all formal-proofy on it
b.) the software people realized that all the specs and architecture documents and fuzz testing and automated semantic analysis and literal actual voodoo dolls all did totally help, BUT...
(you know, they help ... BUT probably wouldn't have helped in this case, or lots of other cases...)
So this is just the way software is in almost every walk of life. ZFS is amazing, one of the most life-improving software technologies for me; up there with antibiotics, and UTF-8.
Yes, they had a pretty amazing bug, too. So yeah, "... at least for now, there's simply no way to reliably detect bit rot and other data integrity issues and be assured they can be remedied."
Which is just like it ALWAYS HAS BEEN and ALWAYS WILL BE.
But in the fullness of time, so far, ZFS has gotten us closer than anything else to that (impossible, unacheivable) ideal.
Yes, agree. However, I also lost files on it (had files overwritten with 0s, as other people already commented here). Also, it's your best bet of you don't need the extra features provided by ZFS. Thanks to its checksum validation, I detected bad SATA cables three times already (on different systems), instead of getting corrupted data.
>it was beautifully designed and implemented by SGI back in the day. Hopefully the ChatGPT generation don't start modifying that code too much.
Except it's not what happened. Please check this talk from Dave Chinner, where he explains how XFS was developed and ported to Linux and then made robust and stable by him and other contributors. The disk format has changed many times, now it's robust and resilient against metadata corruption.
XFS Development talk by Dave Chinner https://m.youtube.com/watch?v=FegjLbCnoBw&t=683s&pp=ygUQeGZz...
History of Linux FS, also by Dave Chinner: https://m.youtube.com/watch?v=DxZzSifuV4Q&pp=ygUQeGZzIGRhdmU...
But to my understanding, these are 2 different things.
- ZFS is the very stable, non open source project that only works on BSD. - Open ZFS is an effort to rewrite ZFS in a totally open and free format that now runs on Linux.
This was the idea I had, but I might be incorrect.
Open source ZFS wound up in FreeBSD (and from there, other BSDs) before Sun stopped releasing Solaris source.
ZoL was a project based on said open Solaris source to run it on Linux, and while the Linux glue layer was not the FreeBSD glue layer, it was still based on the same ZFS source.
OpenZFS switched from being based on the illumos codebase to being based on ZoL a couple years ago, merged in FreeBSD support, and FreeBSD 13+ ships based on OpenZFS and not the original ZFS port to FreeBSD.
None of them are a new implementation.
Talk by Bryan Cantrill https://www.youtube.com/watch?v=-zRN7XLCRhc
44:16 "[paraphrasing Bryan C.] I told Oracle management if they changed the license, the core ZFS would quit. And they lost all the ZFS team."
You can look at ZFS for inspiration. It is more complex than say ext4, but it's less complex than the full stack of mdadm+LVM+ext4 - the latter piles up the abstractions, where ZFS is able to "reach around" and e.g. directly track free physical blocks across multiple devices. So when you need to resilver the pool, you don't need to copy the unused blocks, and reduce the load on the pool (and thus the chance of double failure).
Are there programming languages that have such properties? At the risk of perpetuating the meme, I'd say Rust (with no_std) fits that description. I can't tell how would it help in this specific instance (I'm very far from an expert in FS implementation), but it does tend to prevent data races in general.
At that point, is Rust buying you anything for how much it's going to get in your way?
I really don't see an advantage to Rust when operating at these kinds of low levels.
And beyond that, rust has many features that are useful separate from memory safety.
https://asahilinux.org/2022/11/tales-of-the-m1-gpu/ being just one example.
That said I have no opinion if they should write this driver in Rust or not, I simply do not know about the details. But in general, “it’s too low level and so tons of unsafe and so therefore Rust is useless” is at least arguable, if not just fully incorrect, as a general point.
The complex stuff can and should be written at a higher level than C.
Yes, I have. And I have implemented partial file locking semantics. And I can painfully remember what I went through to validate it.
Quite a few of those pointers are write pointers which are simultaneously active with a lot of read pointers and they have different owners. That is a task which is screaming "Rust is going to make your life miserable."
The next step up for that is formal verification which is staggeringly time consuming (read: expensive) and hard to understand in the first place.
But that would make it way harder to add to systems it's already running on.