Hacker News Reader

Comment by pantalaimon | Hacker News Reader

I was very excited about btrfs' advanced features, but that meant that btrfs would bite me multiple times when I expected it to 'just work™':

- RAID5/6 are still not stable

- it will not mount a RAID in degraded mode automatically, failing the high availability promise that might tempt you towards RAID.

- swapfile support exists, but it breaks snapshots (and I don't want to snapshot the swapfile)

- Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists)

View on HN

> it will not mount a RAID in degraded mode automatically,

I remember when btrfs was very young and they announced the ability to create mirrors. I tested this out and was pleased, and then I tested the failure scenario: pull a drive, try to boot.

It wouldn't boot!

I jump into IRC and ask if it's expected that you can't boot from a degraded mirror and the answer was "not supported" which means the mirror is pointless.

Obviously it has improved since then as there's a way to force it to work, but I returned to ZFS on FreeBSD and never looked back

Yeah this is the probably the worst default for a fs that supports raid.

It's the safest default. If you want a filesystem that automatically downgrades the safety of your data when things go wrong, you should have to opt-in to such behavior (and btrfs does make that possible).

There's less risk of damage from running off a single disk in a faulted mirror than to rebuild the mirror. The stress the drive is under to rebuild can and will kill drives at the end of their life.

If one half of a mirrored pair dies, you're left with one copy of the data on a drive that's at risk of also dying, plus whatever portion of the data you have backed up. If you then write new data to that drive, it isn't mirrored anywhere and is also guaranteed to not be backed up yet. So the newly written data is at equal or higher risk than the data that was already on the surviving drive. It's not hard to see how silently accepting new writes that cannot be mirrored could be an unacceptable risk to some end users.

With btrfs, you can easily add a new drive so that new writes can be accepted and mirrored immediately, then start rebuilding the old data with lost redundancy (possibly after running another incremental backup, which will be less stressful to the drive than the full rebuild). But the "add a new drive" step happens outside the kernel, in userspace and possibly in meatspace, so it can't be a default action for the kernel to take.

If that is a concern, then it sounds like running mirrored disks isn't sufficient for your use case and maybe require three disks to enable a disk failure and a rebuild to not cause an issue.

Also, if you're happy running off a single disk in a faulted mirror, then I'd question why you've got the mirror setup at all.

- RAID5/6 are still not stable

Why does everyone insist on RAID5? I am completely fine with btrfs RAID1, survived few disk crashes as advertised.

- it will not mount a RAID in degraded mode automatically

it will if using -o degraded, it's the tooling which generally sucks and won't support this. I even had problems booting at all with / on multi-device btrfs using recommended tools (dracut, grub-mkconfig), the bugs are known and unfixed, ended up rolling out own initramfs.

- swapfile support exists, but it breaks snapshots

you are supposed to have all your stuff in a subvolume, and snapshot that, not whole toplevel root filesystem...and yes it should be documented better that it interferes with snapshots

- but snapshots are not integrated..

yep tooling sucks

RAID1 is a lot more expensive and/or stores a lot less data than RAID5/6.

How much is the "a lot"?

You can try yourself: https://carfax.org.uk/btrfs-usage/

But generally, RAID1 gets you 50% of storage space, whereas RAID5 gets you 66% (and any odd combination of disks).

RAID5 should actually get you n-1 disks of space (if they're equally large), while raid1 only gets you n/2

Raid1 actually gets you less than that. If you mirror 5 drives that only gives you 1 drive worth of data (with 4 redundancies).

But… you have to weigh it against losing all your data. With raid5 if any 2 of your disks die at the same time then you've lost your volume and most likely 100% of your data (no matter how many disks you have).

With raid1 you have to lose _all_ your disks before that happens. Typically that's also just 2 but you can mirror 3 or more drives if you need some data to be _really_ resistant to disk failures and you don't care about "wasting" n-1 times the space.

So you end up trading off efficiency of storage space and resiliency to data loss.

Myself, disks got cheap enough that I always just buy 2 disks and mirror them. I find it easier to reason about overall, especially in the face of a degraded array.

We're talking about btrfs raid1 here which always gives you half of total capacity of all disks (except for extreme cases). With 5 drives it's 2.5 drive worth of data. If 2 drives die means 1 drive worth of data was lost (more or less, depends on balance).

> Just an Ubuntu/Debian thing, but snapshots are not integrated into the update process unless you install `apt-btrfs-snapshot` (and know that package exists)

Thanks - I did not know about that.

I agree with the stance of not mounting a degraded RAID automatically as then the danger is that someone might not notice it and be subjected to total data loss later on. The best option would be to allow over-riding that choice if the RAID is otherwise monitored.

Can you not just place the swapfile inside of another subvolume? Since subvolumes are not included in snapshots. There's generally a bunch of stuff in /var that you don't want to include in snapshots, too, so it's not like putting the swapfile inside of a subvolume is an exotic task.

I guess so, I just didn't know when I created the swapfile and was surprised later when I tried to create a snapshot.

Be careful, just like disk images, you probably don't want to place swapfiles on a CoW filesystem.

You can mark individual files as No_COW in btrfs, and No_COW + preallocation is a requirement for swapfiles anyway due to how the swap subsystem works.

Reply on news.ycombinator.com