Hacker News Reader

Comment by boris | Hacker News Reader

>Snapshot implementation has been started, but snapshots are by far the most complex of the remaining features to implement

Snapshots are the #1 feature of COW filesystems. I've been using them for a bit in btrfs and this feature is game-changing (and no, it hasn't eaten my data yet).

View on HN

At first glance, the status page of btrfs look horrible:

https://btrfs.wiki.kernel.org/index.php/Status

The problem areas are mostly RAID and exotic features. RAID can be handled by a different layer and most users don't really need the exotic features.

Judging from the media silence in the last months I'd say either people stopped using btrfs or it just about works good enough for everbody.

when the RAID is handled by a different layer you lose some very important integrity features

IIRC, doesn't BTRFS also allow you to do cool things like change RAID levels dynamically? (E.g. You can be running a 2-disk RAID-1 array, pop in another disk and tell BTRFS to make it a RAID-5 array instead, then a year later pop in 2 more disks and switch to RAID6, all with no downtime.) I imagine that wouldn't be possible if you were doing RAID at a different layer.

Linux's built-in software RAID (implemented at the block level) has supported online RAID level changes for many years. Check out the "grow mode" section of the mdadm(8) man page sometime; it goes into great detail about which operations are supported.

Yeah, shout out to mdadm and the md driver. I've been using it for years and years and it's rock solid. Being able to grow arrays (online) and convert plain drives into mirror sets is great. I feel that md and lvm are underappreciated by a large number of Linux users…

That's why all big data in big corporations uses exotic-open-source-filesystem-based RAID. That's where the action is when it comes to integrity.

None of that virtual block device or driver-level software junk, let alone hardware RAID controller solutions.

Well, there are other reasons; you want to write code that operates on the data, and neither the code nor the data fits on a single machine - you have to target an abstraction which spans machines. Block storage is too low-level an abstraction.

That isn't to say that using high performance block storage isn't still a win even when the redundancy is multiplied at a higher level. The higher level redundancy is also about colocating more data with the code - i.e. it's not just redundant for integrity, but to increase the probability it's close to the code.

Block storage can be network-abstracted.

Even virtual memory for that matter. Now ancient concept:

https://en.wikipedia.org/wiki/Distributed_shared_memory

Of course. Most production monoliths are deployed on networked block storage - aka SAN - and NUMA is already structurally distributed memory, even on a single box. But it's not the right paradigm to scale well, no more than chatty RPC that pretends the network doesn't exist is the right way to design a distributed system.

I think the high profile ones (Google, Amazon etc) use relatively dumb OS drivers and do the fancy distributed FS abstraction stuff in userspace. Certainly stuff like Ceph and Gluster don't have very good reputations and are mostly sold to relatively clueless "enterprise" customers.

RAID 5/6 in btrfs doesn't have data integrity features so you're back to square 1 on this regard.

OpenSuSE uses btrfs by default and relies on it for one of it's killer features. Before a system change, such as installing updates, changing services configuration, etc, is made using YaST, snapper takes a snapshot of the root filesystem. If something breaks, just roll back to the previous working state.

I'd say that for the OpenSuSE folks, btrfs falls squarely into the "good enough" category.

I'm a big opensuse fan, but none of my opensuse machines are running btrfs.. Although, besides the stability (which can't really be much worse than ext4/xfs which is what I apparently choose on the two machines) I think what drove me nuts about opensuse's use of btrfs was all the subvolumes. Which are cool, but just another thing for me to deal with, and my computing theory for the past few years can be summarized by "KISS, unless its really hurting".

FWIW, Solaris has been providing similar upgrade safety via ZFS since 2008.

Sadly, I got out of the Solaris game a couple of years before ZFS became a thing. I'm using ZFS on FreeBSD though, I quite like it.

Yes, FreeBSD adopted the same command Solaris has for boot environment management -- see the beadm man page.

Yes, operating systems other than OpenSUSE have this feature. TrueOS (formerly PC-BSD) has had it for some years, too.

* https://blog.pcbsd.org/2012/07/9-1-feature-multiple-boot-env...

* https://blog.pcbsd.org/2013/06/pc-bsd-status-update/

* https://www.ixsystems.com/blog/a-closer-look-at-the-changes-...

* https://www.ixsystems.com/blog/the-revamped-life-preserver/

If you're using large raid/jbod arrays then the chances are you're using zfs or hardware raid.

I wouldn't want to be the poor sucker supporting a large BTRFS array.

> Judging from the media silence in the last months I'd say either people stopped using btrfs or it just about works good enough for everbody.

3rd option: there aren't really much people who ever used btrfs.

from the feedback I could gather when I enquired about it, it's 33/33/33:

33% who tried and say it's fine

33% who tried, hit a few bugs and stopped

33% who say it's known for not really being finished, not a good idea to go for it.

This is kinda a subjective thing though. For me the #1 feature is being resilient to bitrot.

if you disable cow, you also disable checksumming, so they are kinda linked

"but snapshots are by far the most complex of the remaining features to implement"

I thought defrag (or things like defrag) was the most complicated thing to implement ?

ZFS has always had snapshots but I am told defrag is a long, long ways away ...

According to Kent on Reddit, bcachefs technically had online defrag since it was just bcache -- via the copying garbage collector for reclaiming space. So bcachefs will simply inherit that feature by design, which is great.

Apparently copygc is off right now because reasons, though (I'm going to assume it's almost certainly the related extent/compression issue that's holding this up from being enabled, which you can see referenced on the home page, at the bottom).

I should have been more clear - copygc is off by default in upstream bcache, it's on in bcachefs (and required, in order to guarantee a capacity when doing random writes)

Reply on news.ycombinator.com