A detailed look at Ubuntu’s new experimental ZFS installer
arstechnica.com
arstechnica.com
"By the numbers: ZFS Performance Results from Six Operating Systems and Their Derivatives" https://www.youtube.com/watch?v=HrUvjocWopI
Slides: https://callfortesting.org/log/vBSDcon-2019-Dexter-BytheNumb...
And was surprised that zfs for windows (and hyper-v core) appear to be alive and comming along.
Anyone using zfs on windows? Are they also going off of zfsonlinux - and if so how's support for encryption comming?
Finally having shared, read write, encrypted home that works on both Linux and windows safely would be great!
I'm really keen to give ReFS+SS a go, but I keep hearing horror stories of abysmal parity-spaces performance, which can only be explained by poor coding and testing.
Even if ZFS has better performance, it's never going to have full support, and it wasn't written for the NT kernel IO model, so I'm dubious about it ever working well...
Sincerely, someone who had to port kernel level code from Linux to NT.
I quickly moved away my data and never used ReFS since.
Which is almost more interesting than all the good stuff that comes with zfs (volume management, disk management, software raid-like functionality, encryption, snapshots, send/receive).
Its the first time we truly have a half-modern filesystem with safe write support and sane performance across many different os's. Even if just made dual booting freebsd and Linux this would be a fmgreat win imnho.
If the choice is between zfs and fat32...
IIRC, one of the issues involves the layout of extended attributes. FreeBSD keeps to Solaris layout (which is essentially the same in concept to NTFS' one), while Linux for speeding up certain stuff (and because Linux can't use native xattrs to full extent anyway, due to using Irix API which is limited to PAGESIZE xattrs) has an optional feature to pack xattrs more densely.
Sounds like you probably ran into zfs not being in Debian proper - and buster unlocking dmcrypt, but not being able to mount zfs? Should normally be salvageable via booting previous Jessie kernel and initial ram/root disk (initrd).
At any rate, I gather you've moved on - just sound like an unlucky break - even with root on unsupported zfs.
But then again, the main reason I haven't is because all of my stuff is backed up via Nextcloud, so reinstalling Debian is pretty quick/painless for me.
Zfs encryption was still a bit new, so both boxes uses dmcrypt on full disk plus efi plus /boot (technically redundant, but it works).
Looking forward to hopefully root on encrypted zfs in next Ubuntu lts.
Curious if snapshots can protect you even as far as OS upgrades gone wrong. I'd assume they can, if all filesystems are ZFS + snapshotted.
My issue was that for some reason Linux no longer could find the encrypted partition to decrypt, so I couldn't get it up to a point where I could even work with it.
Your experience has definitely not discouraged me from wanting to ZFS my root partitions though. If only to provide the potential / theoretical ability to roll back a snapshot if necessary!
I can post my "script" when I get home if you want to try it. I use script very liberally, it was my notes of the commands I used in order in order to get ZFS/dmcrypt working.
(open) Zfs now has native encryption:
https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0
(ed: there are some potential benefits to zfs encryption - it's as cross platform as zfs (encrypt an USB drive, use on Linux, os x, windows, freebsd).
Zfs send/receive - receiver can take your encrypted backup - key doesn't have to travel.
Mount drives across dual boot - share encrypted /home between dual boot windows and Linux?
Not sure - but I think volume encryption is supported too. Might simplify encrypted swap (if you already "pay" for zfs). Note: there are apparently some issues with swap on zfs (volumes at least, possibly files too)
If you don't need suspend/resume - suppose you could share one encrypted swap volume across multiboot environments..) )
Why?
One more addition to other replies: ZFS needs access to disks for self-healing (bit-rot detection and repair). It means that in the past if we wanted encryption, same date needed to be encrypted twice (one for every mirror).
Now with encryption in ZFS, date is encrypted before creating redundancy. We can even do a scrub without encryption key.
Encrypted /home/username filesystem with user-owned keys, is one neat trick,for example.
The ZFS specific encryption IIRC is owned by Oracle, and they do not open source it.
Instant backups (snapshots) alone are worth the investment. Take a snapshot before you update your kernel, and poof super-simple recovery. Working on a project and need to access it remotely? Take a snapshot and then pull it to wherever you are, your dev environment anywhere. (Different use-case from GIT.)
Snapshot /home/ and worry less about computer migrations. ZFS is _THE_ power-user tool of the decade hands down.
Disclaimer: I run FreeBSD on my workstation, laptop, and servers. I've never tried to "mix & match" FreeBSD & Linux ZFS implementations.
I use it mainly for non-root volumes because of bad experience in the past (dkms deciding to not build zfs module on first try after update, kernel incompatibility, grub incompatibility, etc.). It's better now, but unless you are experienced I would suggest to not use it for root.
Main reasons I use it:
* checksums - I had undetected data corruption
* snapshots - great addition to backups
* compression
One thing if you start playing with it (I am surprised how little users know about this feature): you can access the snapshot content using hidden "directory" /.zfs/snapshot/snapshotname ;) Just cd into it.
One word of warning, near end of pool capacity it will start to fragment (there is no defragmenter for it, but if they have made implementing encryption, defragmentation shouldnt be far away) and performance will drop. Dont let it happen, always keep 10-20% free and dont forget about old snapshots size.
The only thing is that it "uses" all your RAM, but gives it back as needed. Also, if your volume gets over 90% full, it starts becoming VERY inefficient.
I’m running FreeNas at home with 6x4TB WD Red drives. All my computers back up to FreeNas. My Plex server reads/writes all its media from it as well.
I’m going to switch FreeNas to a Linux server with ZFS at some point.
Checksums, compression, snapshots (so easy and fast incremental backups using send/receive).
ZFS is much more then RAID. Also, even with only one drive we can increase our chances a bit using copies=2.
Data validation, snapshots, compression, zfs send (for backups)?
Unleas you're into either file systems or installers as a hobby, let others be the early adopters.
I guess it saves battery due to ARC (adaptive replacement cache).
> Unleas you're into either file systems or installers as a hobby, let others be the early adopters.
ZFS is already quite old filesystem as has great features (for example: checksumming, snapshots, compression) that should be already standard.
Sure but if you don't know how zfs works do you really want to figure it out in the middle of trouble shooting installation problems, or post install root volume problems?
Isn't it similar to using mdraid, lvm and dm-crypt without experience?
> or post install root volume problems?
In my other comments I suggested to not use ZFS for rootfs unless someone is already experienced.
Those are all pretty different, but yeah basically we agree on this. I'm not sure about battery life, if I had to give a more nuanced answer I would say it probably depends on use case and you should measure it if you want to be sure.
I'm not in the home NAS building sphere, but wouldn't you rather have your root be a SquashFS image on an SD card or similar, with the storage section being ZFS?
As for my laptop, I guess the snapshots are nice. I've never had a situation with my laptop where I wished I had snapshots, and even if something did break I only care about the files I have, which I back up. Then the snapshots are an extra safety net I suppose.
The first is generic: I work only on filesystems with data checksumming; the current ones are BTRFS and ZFS, and I have a low opinion of the developers/development of the former.
The second is specific: I keep my disks encrypted; since v0.8, ZFS has encryption integrated, which allows me to have disks encrypted and mirrored without additional layers (BTRFS requires LVM for this setup).
In the past I've also used snapshots, by the way, which are really functional.
Scrub is not even required for that: all data is checked when read, it's only for checking unused data.
> It lets you resize the dataset
Considering that you are talking about quota on directories, I'm not sure it is really advantage of ZFS. It's basic functionality.
There's no real increase in power usage except for one thing, which is configurable - the transaction group commit frequency (I leave it default, a friend of mine sets it to very powersaving level coupled with running "nosync").
A quick description is that TXG write period says how often ZFS writes data that was written without syncing to disk, and commits data that was written synchronously. Every "transaction group" is essentially a snapshot, and by default it's done every 5 seconds, or every ~10? MBs written, whichever is first. Data that is written synchronously between that is written into "ZFS Intent Log", and all data written is kept in memory without sending to disk (except for sync writes).
If you disable sync writes (datasets mounted sync=none), ZFS will only wake up drives for writing with the interval set in this parameter, or when you start writing significant amounts of data (where "significant" is also configurable).
If you're willing to lose X seconds of writes, you can configure sync=none and transaction writes to happen every X seconds. My friend does that, myself I prefer my data safer :)
That's the story on power usage with ZFS. I'd recommend maxing out memory where possible, but I recommend that even without ZFS.
Then there are benefits, like recovery from snapshots (more than once I recovered a file I accidentally overwrote because I missed a '>' in '>>'), easy backup/migration with zfs send/recv. I also found it pretty easy to install multiple linux distributions using the same root pool, by giving them different root datasets and putting home directory and similar on separate dataset (on the same pool).
This allowed pretty nice migration between distributions, so long as I kept UID/GID of my user the same between them.
Read-only root filesystems are one of those things that everybody ought to do, yet are usually impractical. Unfortunately, most software doesn't really support the Linux Filesystem Hierarchy Standard, so you have software that modifies itself under /usr (supposed to be read-only), ever-growing logs and caches not stored under /var/log and /var/cache so they can't be managed automatically, etc.
You generally find squashfs roots where the distro is specialized and not general-purpose - for a recent example, see Talos.
Anyway, the fix is pretty obvious: put each application in its own namespace, thus its view of "/usr" isn't the same as the immutable OS "/usr" and it is free to write whatever it wants without causing any trouble.
Well, that's exactly the job of competent package managers, to make sure that packages fully stated their dependencies (so that the package manager would forbid you from getting into an illegal state wherein a required dependency was not installed or the wrong version of the dependency was installed), and installed correctly according to the LFHS so that there would be no conflicts.
/opt isn't new. Plenty of people decided that they could throw everything into a folder under /opt, including copies of all dependencies, with no references outside of that folder, and that would be enough. And it might have been enough if it wasn't for sysadmins in charge of uptime tearing their beards out over /opt constantly filling up and causing multi-product outages when it did.
The whole point of the LFHS is that it represented a contract along the boundary between developers (who needed to bundle libraries, save state, have access to a cache, etc.) and operators (who worried about disks and associated costs - why would you ever spend a dime backing up /var/cache? Why would you ever use a fast disk for /etc?). When this contract was followed, systems worked splendidly. When the contract was broken, the people who broke the contract shot themselves in the foot just about the same as anybody writing C code knows how many footguns are in the C language.
Of course, today, with the price of resources having crashed through the floor, and containerization having more or less reached maturity, this is all more or less an academic, historical, moot point. But dynamically linked libraries weren't stupid, they were an artifact of an age where statically linking libraries was horrendously uneconomical and the idea of namespacing anything was enough to get laughed out of the room by people wondering how you would expect to pay for any of it.
Package managers are an over-engineered solution to a problem that only really exists if you rely on package managers. Ever had your apt database break on you? It's a joy, let me tell you. Nothing ever gets installed or removed again. Blowing up your system and requiring your OS to be reinstalled is ridiculous. Especially since that was a problem that didn't even used to exist. OSs were once stored on roms.
> /opt isn't new. Plenty of people decided that they could throw everything into a folder under /opt, including copies of all dependencies, with no references outside of that folder, and that would be enough.
Sure, as long as you never move it from opt. UNIX devs' addiction to hard-coding fixed paths is difficult to break.
> And it might have been enough if it wasn't for sysadmins in charge of uptime tearing their beards out over /opt constantly filling up and causing multi-product outages when it did.
Is that really a problem outside of Linux, which famously defined its base-system as "the kernel" and therefore necessitated that any such install would need to bring along all its own dependencies because it couldn't rely on anything existing except the kernel?
> Of course, today, with the price of resources having crashed through the floor, and containerization having more or less reached maturity, this is all more or less an academic, historical, moot point. But dynamically linked libraries weren't stupid, they were an artifact of an age where statically linking libraries was horrendously uneconomical and the idea of namespacing anything was enough to get laughed out of the room by people wondering how you would expect to pay for any of it.
Yet we still live with this bullshit today anyway and the Linux community especially eschews any attempt to move on. This despite some of the same people who created UNIX creating Plan 9, which rejected dynamically linking and embraced namespacing over twenty years ago!
this only happens if one isnt aren't managing applications coherently (e.g binary only packages managed haphazardly, like most linux binary packagers)
on systems where there is a clear base+add-ons distinction (e.g. source based ports), these things happen much less often - all packages are built from a coherent source tree, and read only root is often feasable because packages only install reference files into their respective '/usr/share' area and don't muck with the 'real system' in /etc - the latter being an administrative action (e.g. package installation is a separate action from service configuration / execution)
The OS should be a platform. I should be able to build an application and distribute it to users and have it just run on that platform without involving third party middlemen and regardless of other software they have. You know what OSs accomplished this? The original Macintosh, RiscOS, DOS, and (mostly) Windows 3.1. Why is it that we can't accomplish today something we did 30 years ago?
a platform for 'building your system', or a platform for 'distributing and executing binaries'?
the binary only distribution mentality comes from microcomputer culture, which postdates unix/mini culture and is rooted in non-technical users installing purchased 'application' programs from vendors rather than computer users (read: programmers) building their systems to suit their own personal data processing tasks. you highlight this in your choice of platforms 'getting this right', but really, these are two different mentalities, cultures, and modes of operation.
not saying one is superior to the other, but square peg, round hole.
and besides:
many proprietary programs for unix just ship with static executables. problem solved.
These are not mutually exclusive goals.
> the binary only distribution mentality comes from microcomputer culture, which postdates unix/mini culture and is rooted in non-technical users installing purchased 'application' programs from vendors rather than computer users (read: programmers) building their systems to suit their own personal data processing tasks.
"Non-technical". Elitist bullshit. Woz was non-technical? He was clearly a micro guy. People got work done with micros. They made things with micros. Like your glorious programmer master-race, they configured their environment into something that enhanced their lives. They bought and used software the way you import 10000 dependencies. Excel is still by far the most used programming environment in the world, with entire businesses run on tooling never touched by programmer-master-race hands.
Besides, programmers should want it this way too. Which makes more sense to you? A constantly mutating state where changes in seemingly unrelated portions can wreck havok on the whole, or a collection of immutable components which can be swapped in and out at will? A system so complicated it needs sophisticated management software and a database of conflicts and dependencies to keep it all running, or one so simple that 'copy' and 'install', and 'delete' and 'uninstall' are synonymous?
If the UNIX model is so damn great, why the fuck does Docker exist!? If development is so damn great on UNIX, why are people building and distributing dev environments in containers!?
> you highlight this in your choice of platforms 'getting this right', but really, these are two different mentalities, cultures, and modes of operation.
> not saying one is superior to the other, but square peg, round hole.
I disagree. I think one is clearly objectively superior to the other.
> many proprietary programs for unix just ship with static executables. problem solved.
Ha! Yeah. Problem solved. Even for proprietary software with a native Linux version I've almost always had better luck running the Windows version under WINE.
My main threat scenarios are hardware failures and encrypting trojans. Snapshotting (if done right) protects also against the latter.
Boot environments are very handy for updates:
* https://ramsdenj.com/2018/05/29/zedenv-zfs-boot-environment-...
* https://mwl.io/archives/2363
Also, being able to snapshot the entire system at regular intervals, show diffs between snapshots, and then perhaps do security analysis on the results might be handy.
probably everyone if/when ubuntu decides to make this the default, which they probably will, because `sudo apt install coolpoints`
I loved the flexibility of BTRFS in this regard - you could change the duplication level at any point. BTRFS seemed to abstract the storage away beautifully - just give us your drives, any combination of sizes, tell us how many copies of your data you want to keep and we'll handle the rest. It wasn't perfect, but I loved this approach, and I feel like it could go even further, eg. calculate risk of failure probabilities and then just tell it how low you want your risk. If it can't do it with your current drive capacities it could tell you what it needs to accomplish it, and then rebalance the data once you add the drives. Got an SSD? Throw it in the pool and let it optimize for cacheing. I know bcachefs has that particular aspect covered, but there's a lot of other basic features that still need to be implemented.
And there is work on it for OpenZFS: https://www.youtube.com/watch?v=Njt82e_3qVo
for zfsonlinux As far as I understand you can currently add and remove striped/mirror devices, but you cannot remove raid-z vdevs (ref: https://github.com/zfsonlinux/zfs/issues/9129) - so there is definitely work left to be done in this area.
as an aside mdadm has surprisingly fantastic features in this area and can reshape pretty much any array type to any other. i was impressed by that.
Btrfs has a bad reputation because to many people lost data, including me. I would never again use Btrfs, only ZFS.
Give it a try, it's part of the kernel and it works really well.
When it was eating my data it was also supposed to be stable. But turns out it wasn't.
If you want to use btrfs go ahead, I will not do so again.
Since then all other devices made or officially supported by Jolla are using EXT4 on LVM, including this one Xperia X running Sailfish OS I'm typing this on.
(And yeah, those issues were how I found out: unable to create files while having "4 gb of free space" - not helped by the fact that it didn't run the balancer automatically.)