Proposal to move CoreOS off of btrfs
groups.google.com
groups.google.com
You'd think they have safeguards to automatically rebalance on certain conditions (metadata filling up, etc) but no, you have to babysit btrfs.
I get those, and am not using btrfs...
It's been how many years now waiting for the next great ZFS competitor? If nobody is able to improve on ZFS, how about we just all jump on the bandwagon and move on with our lives?
If nothing else, having more people using ZFS may inspire someone to actually improve on it. BTRFS has such a limited feature set in comparison to ZFS that with the advent of ZFS on Linux, it's really hard to understand the community's continued backing of BTRFS as the Linux community's CoW filesystem competitor to ZFS. Spend a few weeks using ZFS on Linux on a machine with a few SSDs and a few HDDs, or just enable it on a linux laptop with a SSD drive, and it's hard to go back to anything else.
"ZFS is licensed under the Common Development and Distribution License (CDDL), and the Linux kernel is licensed under the GNU General Public License Version 2 (GPLv2). While both are free open source licenses they are restrictive licenses. The combination of them causes problems because it prevents using pieces of code exclusively available under one license with pieces of code exclusively available under the other in the same binary. "
1 - We can't just take the code and add it to the kernel. 2 - The patent and other legal grants in the licence won't apply if we try to reverse engineer a compatible GPL2 equivalent for the kernel to use, which will be a HUGE risk for the few companies that would probably consider the cost of reverse engineering worth it. And before you say it can be done without company support. I have 2 things to say, hows that going for the ReiserFS 4 fans who wanted to keep improving that, and in order to continue the work on ZFS effectively, the existing ZFS developers have 'ganged up' and now try to work on everything so that the openzfs project can be a single source of truth for the ZFS source code, and a reverse engineered version would be very unlikely to benefit from this, and would therefore require even more developer time just keeping up with the improvements 'upstream' in ZFS from the 'original' ZFS codebase.
In the case of the kernel, this prevents us from distributing ZFS as part of
the kernel binary. However, there is nothing in either license that prevents
distributing it in the form of a binary module or in the form of source code.
ZFS cannot be added to Linux directly because the CDDL is incompatible with the GPL.
ZFS can, however, be distributed as a DKMS package separate from the main kernel package.
So why not just distribute the DKMS package with a distro? Easy enough. Will it ruin the user experience somehow?
I wish it were feasible to get CoreOS (and Docker, for that matter) to add support for ZFS. I use LXC + ZFS every day and it is stable, reasonably fast, and all around wonderful!!!
sigh.
It's too bad, I really love the idea of btrfs, but there's no way I would ever run a filesystem on it yet.
When it goes wrong (about every 4 months for me), you will end up in a nightmare. Generally attempts to fix issues will make things worse, various tools and pages contradict each other, and the devs are only interested in the latest kernel version. Much of this is deliberate - the code is written to not sweep things under the rug, which means you can hit problems and not recover. Use backups and make sure you can restore.
The reason why I keep using it is because there is no silent corruption as you get with ext4. A scrub can verify every byte of data is unaltered and recover if using anything other than single profile. Compression, volume management, cheap snapshots etc also make working with it nice. Until things go wrong.
The single biggest "going wrong" is running out of space. Copy on write filesystems by their nature leave existing content alone and write new information in the spare space, eventually doing a garbage collect of obsolete data. When you are out of space that gets rather difficult with bizarre symptoms and tricky recovery.
You can use tools like LVM and md as a layer underneath ext4 to provide some resiliency, but there is a learning curve and two sets of tools to work with. Changing around disk/partition sizes isn't much fun with them.
https://btrfs.wiki.kernel.org/index.php/Balance_Filters
https://btrfs.wiki.kernel.org/index.php?title=Main_Page#News
Just as BTRFS is finally stable and fast CoreOS decide yet again to ship.
We've been using BTRFS in production for over a year now (and heavily with Docker) and haven't suffered any problems at all(1), in fact we actively simulate failures to practice / test repair processes which have all been successful.
To me, while diversity is great CoreOS is going the path of Ubuntu with it's NIH syndrome.
(1) With the exclusion of a slow docker push/pull bug that's about to be patched: https://github.com/docker/docker/pull/9720
Also, it's necessary if they want to switch to overlayfs as the Docker backend, because btrfs lacks whiteout support which causes problems with overlayfs.
Switching to overlayfs for Docker makes a lot of sense, given that you get the dedupe benefits of AUFS with lower overhead than devicemapper/btrfs and it's now part of the mainline kernel tree.
(Interested as I'm thinking about building a homebrew NAS/general purpose server w/ btrfs, there's a lot of outdated info on btrfs but I was getting the impresssion that it's now a pretty stable and useable filesystem)
http://www.nersc.gov/assets/Uploads/W01-ZFS-and-Lustre-June-...
ZFS is meant for managing local drives, and to make it performant you need to configure SSD partitions to act as an L2 Arc cache. The online documentation is pretty good, so after going through the docs it should be pretty clear how to set ZFS up properly for your use cases.
But it sounds like you're using EBS drives? If so, not sure why you'd want to use ZFS. Last I checked, ext2 or xfs was the way to go with EBS drives on AWS. AWS has so much stuff going on in the background to ensure reliability/availability of EBS volumes that adding another layer isn't worth it IMO and I've seen similar kernel panics running other complicated volume managers on top of EBS.
"ZFS NFS" is "linux kernel NFS". Setting "sharenfs" options on ZFS on Linux dataset simply informs the normal kernel NFS service of those exports.
It seems performance have improved and are improving according to this benchmark of last year zfsonlinux by phoronix: http://www.phoronix.com/vr.php?view=19059
Based on past experiences, I don't see Docker-blessed ZFS support coming anytime soon, but I hope I'm wrong. Maybe someone over at Joyent or Oracle can grease the wheels here? :-)