I'll believe it when I see it, why anyone uses BTRFs (UnRaid or any other form of software raid that isn't ZFS) is still beyond me. At least when we're not talking SSD's ;)
ZFS is incredible, curious to mess around with these new features!
ZFS is incredible, curious to mess around with these new features!
That was the day I realized that some open source participants and supporters are interested in having open source projects that are good enough to act as a barrier to entry, but not good enough to compete with their commercial offerings.
Judge the world from that perspective for a while and it can help to explain why so much open source feels 80% done and never gets the last 20% of the polish needed to make it great.
Ooooh. (Booo!)
I wouldn't mind a citation/mailinglist reference for this, if you have one. (I honestly have no idea what I'd Google.)
It's in the quotes from the offline (not on a mailing list) follow up, so it's all hearsay.
I should be clear, I don't necessarily mean I think the developers are complicit in that. I think what happens is more subtle. IE: Companies sponsor the project just enough to be the biggest open source product in the space, but not enough for the developers to make it great.
Or as an alternative conspiracy theory, companies sponsor the projects of great developers that build awesome core features, but never give enough support for someone to turn that into a marketable product. That way they can usurp the work for their own products.
I know there's a lot of speculation there, but, if you watch for it, you can see how most entrenched tech companies are really, really taking advantage of open source developers. Basically the people who are passionate and want to build great things are getting hugely ripped off by people with yachts and rockets.
One of the biggest differences I've noticed between open source and "the cathedral" is that commercial endeavors that revolve around productization tend (as a rule) to manifest sufficient runway to fully round out the implementation of an idea to the point the implementation can participate in the market cohesively by representing itself attractively/competitively. This is often a broad-spectrum effort that requires domain specialization across a huge number of skills, and the burden of sustaining cohesive focus is typically only viable in a commercial context; I think similar levels of adequate collective focus (many individuals, one goal) are only typically raised in cult-type contexts.
Besides the passion-project foundation you mentioned, a lot of open source seems to come into existence because an $employer needed a really specific thing one time and they let the developer license the code under GPL and here it is and there's the 2.8 pages of documentation and it's got some speling misteaks in it and hopefully it works. (...Woops, I just described NPM, and some percentage of PyPI.)
Very very problematically, there's no collective language in the FOSS scene to distinguish between passion projects and commercially-driven JIT-developed code-dumps. After all, the code probably has just as many bugs per 1,000 lines, and the different contexts produce results that work the same, so...?
IMO, being able to encode that attribution to our communication would make SO MUCH difference in terms of user support, project coordination, etc! Coming from a perspective that's still optimistic :), arguing that "this is great, but it doesn't fit our business use case" translates for me to "my contract doesn't extend to me implementing/grokking/mentally integrating/testing/maintaining this new code, and it's not interesting enough for me to figure it out out of hours either" - so the invitation really is there, "send patches in if they're important enough to you", but it requires working cue perception (and possibly lack of cynicism) in all readers in order to be interpreted correctly. IF this is in fact the message that was being sent (!).
That the followup work was not done does indeed waste the effort made by the patch author, and is generally arguably stupid. But this to me brings up questions about the patch author's motivations, and why they didn't have a go at hammering everything into place - because, assuming fully adequate motivation/stamina and sufficient free time to iterate on the patch until the mailinglist likes it, eventually you'll reach a point where either the patch is in a staging tree somewhere, or the list has exploded into a flamewar about why the patch hasn't been accepted already, at which point (continuing to assume ideal circumstances) the patch author could go follow up on all the raised points.
I guess the outcome depends on whether the patch author considers the above AbSoLuTeLy ToO MuCh WoRk SeRiOuSlY ArE YoU KiDdInG Me, or welcomes the community involvement/participation/feedback and does their best to negotiate it to the point of getting the code merged. That the patch author didn't do this is something only they can provide extra context and judgement about; as you noted about speculation, I could only come up with uncited hypotheses here.
Looping back to the first paragraph, there are indeed many instances, for example in the audio/image/video editing scene, where software availability for Linux is incredibly restricted compared to Windows. The options are there, except they don't really work, or they fall over really quickly, or they feel really clunky. I think this sadly comes down to market demand. For example I've tried poking around with simple audio editing tasks - literally just loading a couple of tracks and crossfading them together - on Linux, and come up blank. Pop culture saturation might also be a factor, for example Renderman and Maya have been around for Linux for ages, but it's sliiightly (I suspect) easier to find "how to <jmp> over the license check" thingys for After Effects or Photoshop, for example.
However, with all of this being said, I have noticed a few industries where the comparison of feature parity in what's available in open source vs what's available commercially is sufficiently vast it directly leads to actual mental disorientation ("wait, this is where things are really at??"). It's incredibly difficult in this situation to stay non-cynical and not draw the types of conclusions you allude to in these settings (that perhaps there are agreements in place to not implement certain features, for example). And it's kind of interesting how "get Photoshop" is kind of a thing - an obscure thing, but still a thing that happens, while this seems (generally speaking) to happen less to Eyewateringly-Expensive Software™ targeted at Linux (?).
I guess this reply was me doing the (googles) denial-anger-bargaining-depression-acceptance thing while reasoning through your comment and trying to not be cynical :D, haha. It certainly is one of those worldview classification grey areas, where it's almost like it can be both things at once (except it can't, because that wouldn't make sense)...
I do also definitely agree that a lot of open-source development work is unfairly leveraged, and additionally that the "learning to code" movement (yay, more labor externalization!) is overhyped, almost to the point of the cult thing I noted above.
Because of several things:
* I can mix disk sizes
* I can add new disks over time as needed
* If something dies, up to the entire server, I can just stick any data disk in another system and read it
I didn't want to become a zfs expert (and the learning curve seems steep!), and I didn't want to spend thousands of dollars on new gear (dedicated NAS box and a bunch of matched-size disks).
I repurposed my old workstation into a server, spent a few hours getting it set up, and it works. I've had two disks fail (one data, one parity, and recovered from both). Every time I've added a new disk, it's been 50-100% larger than my existing disks.
I've also migrated the entire setup to a new system (newer old retired workstation), running proxmox, and was pleasantly surprised it only took about an hour to get that volume back up (incidentally, that server runs zfs as well.. I just don't use it for my large media storage volume).
There are drawbacks as the one discussed here, but as a Linux user who doesn’t want to mess up with the FS and uses ZFS for the backup server, the experience has been great so far.
One day I may switch the whole thing to bcachefs, which I've donated and am looking forwards to. For the moment, btrfs will have to do.
EDIT: downvoted by... the filesystem brigade?
I wish you lots of fun with that on btrfs :)
Edit:
https://btrfs.wiki.kernel.org/index.php/Status
RAID56 Unstable n/a write hole still exists
> treated as if I'm storing business data or precious memories without backups, guess I'm just dumb
No your not, but don't use unstable features in a filesystem
The whole point of me using RAID1 (and maybe later RAID5) is that if a disk goes bust, odds are I can still watch a movie from it until I can get another disk. What's more, if I ever fill the RAID1 and I don't feel like breaking the piggy bank for another disk, I can go JBOD as far as my usecase is concerned.
But hey, if the orange website tells me all servers are supposed to be treated as if I'm storing business data or precious memories without backups, guess I'm just dumb. On that note: donations welcome, each 8TB disk costs close to 500 USD here in Uruguay, so if anyone's first world opinion can buy me a couple so I can use the Right Filesystem™, I'd appreciate it!
>so if anyone's first world opinion can buy
Oh buhuu, says the Guy who can afford a NAS for his movies and >2 workstations, stop with your wannabe victim role.
BTRFS has been promising usable RAID5 since 2009 when it was "heading for 1.0" and yet among the most recent developments not but 3 months ago was to add the following warning to btrfs-progs on creation or conversion.
"RAID5/6 support has known problems is strongly discouraged to be used besides testing or evaluation,"
Worse this feature was presented as usable around 2011/12 before being revealed to be unfixably data eating without substantial rewrites in 2016 and 5 years later remains so.
Your hardware might need to be replaced before you can avail yourself of the benefit you posit.
Meanwhile an approach that would actually work on both BTRFS and ZFS would be to add 2 drives to go from RAID1 to RAID10.
The last peel of onion is complaining about down votes. This invites more down votes. If I had to guess people down voted you because you presented a feature that has been a massive pain point for BTRFS as a proposed advantage.
- no data duplicated between page cache and arc
- no upgrade problems on rolling distros
- balance allows restructuring the array
- offline dedup, no need for huge dedup tables
- ability to turn off checksumming for specific files
- O_DIRECT support
- reflink copy
- fiemap
- easy to resize - defragmentationThis is something that has always confused me. BTRFS users are always advised to disable copy-on-write (thus preventing checksumming or compression) for VM images or database files to avoid massive performance hits from fragmentation. Even Facebook still stores its databases mostly on XFS filesystems. However, the ZFS community seems to indicate that you can achieve reasonable performance for databases and VMs just by tuning the recordsize (e.g. https://pg.uptrace.dev/zfs/). How does ZFS mitigate the problems from fragmentation?
BTRFS can do after-the-fact deduplication (with much better performance than ZFS dedup) and copy-on-write files. And you can turn snapshots into editable file systems.
So you clone it and delete some files. All good so far, but the snapshot is still wasting space and needs to be deleted.
But to make this happen, your clone has to stop being copy-on-write. All the data that exists in both /home and the clone will now be duplicated.
And you could say "plan ahead more", but even if you split up your drive into many filesystems, now you have the problem that you can't move files between these different directories without making extra copies.
Jobs never liked anybody other than himself holding all the cards. Having Ellison and Oracle holding the keys to ZFS was just never going to fly.
(Of course, Jobs may have just decided he didn't want to depend on someone else for the MacOS filesystem in any case.)
ADDED: And as others noted, there were also some storage patent-related issues with Sun. So just a lot of potential complications.
I used it for a while but unfortunately since they are not many people working on this and they are not working on it full time it can take them a good while from a new version of macOS is released until OpenZFS is usable with that version of macOS. This was certainly the case a while ago and why I stopped using OpenZFS on macOS and went back to only using ZFS on FreeBSD and Linux instead of additionally using it on macOS. So with my Mac computers I only use APFS.
I’d love to be a fly on the wall for some of those conversations.
https://www.cnet.com/news/larry-ellison-talks-about-his-best...
They could relicense their codebase, of course, but the number of changes that have happened since they diverged is not small.
As others have stated, dtrace has the exact same license and has been in MacOS for years.
While I understand why it would be useful in a corporate setting, for personal use I've found the combination of LUKS+LVM+SnapRAID to work well and don't see the benefit of switching to ZFS. Two of those are core Linux features, and SnapRAID has been rock solid, though thankfully I haven't tested its recovery process, but it seems straightforward from the documentation. Sure I don't have the real-time error correction of ZFS and other fancy features, but most of those aren't requirements for a personal NAS.
That's my case, I'm learning before setting up a cheap home lab and a NAS, and I'm wondering if biting into ZFS is just the best option that I have given today's ecosystem.
It's not terrible, but there are a few new concepts to come to grips with. Once you have them down, it's not terrible.
If you don't plan on raiding, IMO, ZFS is overkill. The check-summing is nice, but you can get that from other filesystems.
Maintenance is fairly straight forward. I've even done a disk swap without too much fuss.
The biggest issue I had was setting up raid z on root with ubuntu was a PITA (at the time at least, March of this year). I ended up switching over to debian instead. Once setup, things have been pretty smooth.
* Checksumming
* As you mention, easy maintenance
* Snapshots and how useful they are for backups
In the end what I value is stuff that works reliably, doesn't get in the way, and requiring minimal supervision. And in the particular case of FS, I'd like to adopt a system that helps avoid bitrot in my data.
Could you drop some names that you would consider as good alternatives of ZFS?
Otherwise it's sort of figuring out what features you want to drop. XFS and ext4 are probably where I'd look for a single disk hard drive.
Like I said, you could do ZFS, but definitely feels a bit like overkill. Setting up a vdev with one disk just to get snapshots and checksums seems like a lot.
But I've recently found bitrotin some of my data files and now that I happened to be learning about how to build a NAS, I wanted to make the jump to some FS that helps me with that task.
Could you mention which tools you would use to replace ZFS? Think of checksumming, snapshotting, and to a lesser degree, replication/RAID.
ZFS is the simplest stack that you can learn IMHO. But if you want to learn all the moving parts of an operating system for (e.g.) professional development, then more complex may be more useful.
If you want to created a mirrored pair of disks in ZFS, you do: sudo zpool create mydata mirror /dev/sda /dev/sdb
In the old school fashion, you first partition with gdisk, then you use mdadm to create the mirroring, then (optionally) LVM to create volume management, then mkfs.
Looking back, the lessons that come to mind are:
- Always have 2 backups (not counting the primary copy), at least 1 "cold" (inaccessible without human intervention) and at least 1 offsite. Backup frequently and retain old backups. With backups, bad decisions are reversible.
- With btrfs or zfs, using a collection of 2-disk mirrors was useful because it provided flexibility (to expand the array, just add another pair of disks) and seemed to have better performance than a single disk. Try to pair disks from different manufacturing batches though. I saw two disks from the same batch and _used in the same mirror_ fail in the same month, which was disconcerting.
- The only data corruption I had to deal with was from RAM that started off good and went bad after a couple years.
- Standardizing on btrfs or zfs from the beginning would have allowed backup by sending snapshots, which would have been a lot easier than cobbling together a solution using rsync.
- Scrub on a regular schedule. Set up monitoring software to notify you of the outcome of each scrub and of any SMART errors.
So I'm starting small, from powering up a ThinkCentre M910 I had laying around, with an internal disk that can be used to store backups. I have 0 need for performance so my idea was to extend storage with an external USB3 HD enclosure. For now, I don't have the space nor the machine where to install dual hard disks for building a decent RAID. Time will tell.
+ your fs
Yeah that sounds like a lot less complexity
Using composable tools is also better from a maintenance standpoint. If tomorrow SnapRAID stops working, I can replace just that component with something else without affecting the rest of the system.
Did you miss the part where I mentioned "for personal use"?
> Since all those tools are from different dev's the system gets more complex.
I fail to see the connection there. Whether software is developed by a single entity or multiple developers has no relation to how complex the end user system will be.
But many small tools focused on just the functionality I need allows me to build a simpler system overall.
The first part of this sentence is probably true, as far as I see, but the complexity of a system perceived by the user depends primarily on the "surface" of the system. That surface includes the UI, the documentation and important concepts you have to understand for effective usage of the system. And in that regard, ZFS wins hands down against LUKS + LVM + SnapRaid + your FS of choice. Some questions a user of that LVM stack has to answer, aren't even asked of a ZFS user. E.g. the question how to split the space between volumes or how to change the size of volumes.
Since ZFS is simpler to use then your setup, is used to store 55PB of data without a single bit error since 2012, i don't see why someone should use inferior stuff, even when it's "personal use".
>But many small tools focused on just the functionality I need allows me to build a simpler system overall.
Sometimes monoliths are better for example the network-stack and storage....maybe kernels (big Maybe here)
Can you actually? If some layer of that storage stack stops working then you can no longer access your existing data, because all these layers need to work correctly to correctly reassemble the data read from disk.
Everytime I looked into setting up a freenas box, every hardware guide insisted that ungodly amounts of absolutely-has-to-be-ECC RAM was essential, and I just gave up at that point.
https://www.freenas.org/hardware-requirements/
I myself use freenas with 16GB of non-ECC ram.
Of course it is possible to have a bit flip in memory that is then dutifully stored incorrectly by ZFS to disk, but this was a possibility without ZFS as well.
I've actually been waiting for this feature for since I first setup my pool. It seemed theoretically possible we were just waiting for an implementation.
ZFS defaults to assuming it is the primary reason for your box to exist, but it only takes two lines to define more reasonable RAM usage: zfs_arc_min and zfs_arc_max. On a NAS type server, I would think setting the max to half of your RAM is reasonable. Maybe 3/4 if you never do anything except storage.
ECC is not recommended because ZFS has some kind of special vulnerability without it; ECC is recommended because ZFS has taken care of all the more likely chances of undetectable corruption, so that's the next step.
For a long running, non-idle system, a good rule of thumb is that all RAM not being actively used is being used by evictable caching.
total used free shared buffers cached
Mem: 12286456 11715372 571084 0 81912 6545228
-/+ buffers/cache: 5088232 7198224
Swap: 24571408 54528 24516880I don't see why ZFS would be more prone to data integrity issues spawning from a lack of ECC than any other filesystem.
If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server, then the argument that ZFS necessitates ECC more than Ext4 or Btrfs or XFS or whatever doesn't really seem to be accurate.
Agreed.
> If I were running a server farm or something, then yeah, I'd probably use ECC memory, but I think if you're running a home server
Then you should still use ECC RAM, regardless of what filesystem you're using.
No, really. ECC matters (https://news.ycombinator.com/item?id=25622322) generally.
https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
(the gist of the scary story is that faulty ram while scrubbing might kill "everything".) However, in the end ECC appears to NOT be so important, e.g., see
FreeBSD Mastery: ZFS by Michael Lucas around pg 174
Deduplication Memory Needs ==========================
"For a rough-and-dirty approximation, you can assume that 1 TB of deduplicated data uses about 5 GB of RAM. You can more closely approximate memory needs for your particular data by looking at your data pool and doing some math. We recommend always doing the math and computing how much RAM your data needs, then using the most pessimistic result. If the math gives you a number above 5 GB, use your math. If not, assume 5 GB per terabyte."
https://www.tiltedwindmillpress.com/?product=fmzfs
This is not to say you need 5GB for every 1TB of data. It doesn't even mean you need 5GB of data for every 1TB for which you have enabled dedup it means you need approximately 5GB of data for each TB of data which is both duplicated and residing on a dataset for which you have enabled dedup. Because of the high memory cost of dedup which rises exactly in proportion to its utility its only useful in cases in which you can plan ahead for its requirements. 99% of users are unlikely to use dedup however this doesn't stop some, not you obvious, from promoting the idea that ZFS requires 5GB of memory per TB or some some absurd figure.
As an aside I really liked the book I found it easy to read and understand and very informative despite being focused on FreeBSD its mostly applicable to Linux as well.
ECC tends to attract zealots after a perfect error-free existence which ECC does tend towards but doesn't deliver, it just reduces errors. I personally don't care about a tiny amount of bit rot (zfs will prevent most of this) and rebooting my storage machine now and then.
You can run ZFS/freenas on a crappy old machine and you'll be just fine as long as you aren't hosting storage for dozens of people and you aren't a digital archivist trying to keep everything for centuries.
Real advice:
* Mirrored vdevs perform way better than raidz, I don't think the storage gain is worth it until you have dozens of drives
* Dedup isn't worth it
* Enable lz4 compression everywhere
* Have a hot spare
* You can increase performance by adding a vdev set and by adding RAM
* Use drives with the same capacity
To add to that, ZFS dedup is a lie and you should forget its existence unless you have a very specific scenario of being a SAN with a massive amount of RAM, and even then, you had better be damn sure.
I really wish ZFS had either an option to store the Dedup Table on a NVMe like Optane, or to do an offline deduplication job.
Now, that becomes the only place entries on it are stored, so you best make it redundant if you don't want to lose your pool from a single NVMe failing, but the feature is there.
The latter I would predict seeing approximately when the sun burns out, on ZFS. It _really_ doesn't like the idea of data changing locations retroactively.
I'm going to have to do some test setups with this.
Is the perf penalty low enough now that it just doesn't matter? I've always disabled compression on datasets I know are going to store only high-entropy data, like encoded video, that has a poor compression ratio.
I second the hot spare recommendation many times over. It can save your bacon.
usually it will hit nothing and have no side effects
I have several file-servers all use ZFS exclusively. and 10x that number of servers using ZFS as the system FS.
Rule of thumb that I like: 1GB RAM/TB of storage. This seems to give me the best bang-for-our-buck.
For a small (under 20) number of office users, doing general 'office' stuff, using Samba, it's overkill.
For large media shares with heavy editor access, and heavy strains on the network, it's a minimum.
Depends on what the server is serving.
DeDUP is a different story. The RAM is used to store the frequently accessed data. If you are using DeDUP you fill the motherboard with as much RAM as will fit. NO EXCEPTIONS! This may have been the line of thinking that scared you away from it.
I have a 100TB server that is just used for writing data to and is never read from (sequential file back-ups before it's moved to "long term storage"). It has 8GB of RAM, and is barely touched.
I also have a 20TB server with 2TB of RAM, that keeps the RAM maxed out with DeDUP usage.
ECC: It's insurance, and it's worth it.
You effectively (unless you use allocation classes) need to keep the entire DDT in RAM all the time if you don't want any write to a dedup-enabled dataset to potentially require blocking on reading the relevant segment from spinning disks into RAM (thus tanking performance even worse than dedup normally does). It's not really related to the mechanisms in the rest of ZFS for keeping {frequently,recently} used data cached in RAM.
Sources: - https://www.reddit.com/r/DataHoarder/comments/3s7vrd/so_you_... - https://www.reddit.com/r/homelab/comments/8s6r2r/what_exactl... - My own tests with around 8 TB ZFS data in a Linux vm with 256 MB RAM.
ZFS only needs a lot of RAM if deduplication is enabled. And it shouldn't be for most use cases, or only enabled on one dataset that benefits from it.
Many ZFS installs are fine on 8GB or less.
ECC RAM is better but not required. The idea is to catch memory errors, hence ECC is better.