Comment by bkor | Hacker News Reader

Comment by bkor | Hacker News Reader

I've quickly skimmed the discussion on fedora-devel regarding btrfs. I wondered mainly how they'd handle the various cases where btrfs does not work well, e.g. files that change often inline (databases, VMs, etc). Apparently an application can tell to treat those files differently. So it's basically a matter of fixing various software to work nicely with btrfs as well as any similar filesystem. As mentioned in the thread, openSUSE already uses btrfs for loads of years. I do wonder why it isn't more supported by upstream software (postgres, VM things, etc).

I was also surprised how good the discussion was on fedora-devel. It seems that people didn't break into camps of "over my dead body", and "if we do not do this I'll leave", etc. It seemed more like a "this is my really thought out proposal, but is it actually feasible or did I forget anything". Then people raised problems, sometimes the proposer(s) already had a solution, sometimes it was something they didn't think of. What's nice is that despite seeing various problems everyone seemed to understand that people are trying to improve things.

I didn't read fedora-devel for many years so that was a welcome change over the past.

> So it's basically a matter of fixing various software to work nicely with btrfs as well as any similar filesystem. I do wonder why it isn't more supported by upstream software (postgres, VM things, etc).

The concept of an application 'supporting a (specific) file system' sounds slightly ridiculous to me.

Why is Btrfs so special that apps should be rewritten for it? The always-compared ZFS seems to work just fine without alterations being necessary, so what make Btrfs worthy of extra considerations?

At most perhaps copy-on-write (COW) file systems as a general class could perhaps be considered, as that may help with things like shingled drives and SSDs (i.e., zoned drives may be aligned with COW FSes).

Because Fedora, among other distributions, will not include ZFS for licensing reasons. (Leaving aside any other discussion of the relative merits of ZFS and Btrfs.)

ZFS has a good share, if not 100%, of the same problems as Btrfs, and in the end of the day it's always matter of leaving files/directories/volumes without CoW, which is hardly a software rewrite as the grandparent insinuated, and more of a packaging issue.

>ZFS has a good share, if not 100%

ZFS has no Raid5 write-hole...no traditional Raid problems at all (z1/2/3)

>leaving files/directories/volumes without CoW

No...just NO, just for a special Cases like a Database.

> ZFS has no Raid5 write-hole...no traditional Raid problems at all (z1/2/3)

The thread was about where CoW is problematic, both ZFS and Btrfs are identically problematic in this sense.

> No...just NO, just for a special Cases like a Database.

Like... for what we were talking about.

Historically programs are "rewritten" (really, adapted) all the time to support changes to the underlying system. E.g., read-ahead, mmap() support, fadvise() support, file locking, different file locking semantics and failure modes in network filesystems, sparse files, hashed directory entries, asynchronous I/O, changes to underlying media characteristics like SSD seek times, etc.

> The concept of an application 'supporting a (specific) file system' sounds slightly ridiculous to me.

This concept is actually way less ridiculous than you might think, for any application which needs any guarantees about data durability, locking, etc (which includes everything from the obvious ones like postgres to things like Dropbox). I found https://danluu.com/deconstruct-files/ a fascinating read diving into this.

To me, it doesn't sound any more ridiculous that applications having specialized support on different architectures. Most applications get away without it, but when performance really matters and general-purpose abstractions aren't sufficient, having special cases seems okay—especially when there's only a small number of architectures (or filesystems) to handle.

This can go too far—I still remember web development a decade ago—and I would prefer to have perfect abstractions that never need piercing, but absent that specialized logic seems _fine_.

> The concept of an application 'supporting a (specific) file system' sounds slightly ridiculous to me.

Clearly you weren’t around in the days when NFS was a thing.

Wait, NFS isn't a thing anymore?

> Clearly you weren’t around in the days when NFS was a thing.

I was and am. I currently help admin an HPC environment with several petabytes of NFS storage.

And the main problems I've experienced with NFS have generally been with Linux locking. I've had a lot less drama with Solaris and BSD NFS (though I did uncover two Solaris NFS server bugs over the years).

Whenever I have NFS issues I generally start with the assumption that there's a bug in the NFS client code.

If you think applications work on NFS by just expecting general Unix filesystem semantics you’re in for a surprise.

The fact you’re not having trouble just means either the application writers put in the effort to properly support it, or your use case is simple enough not to run into problems.

> either the application writers put in the effort to properly support it

Given the hiccups that we've had with MySQL and locking, I'm not sure that's the case. (Less so with Postgres.)

I have only a moderate amount of knowledge of the applications that the dozen research groups use on the cluster, I just make sure the bits flow and haven't heard too many complaints over the years.

Clearly packages like MySQL and PostGreSQL made sure they work on NFS or they would very clearly tell you not to use it because they’d know it’s a recipe for disaster. The issues I am referring to typically are with things like homegrown shell scripts or research software.

>databases, VMs, etc

You should disable CoW and Caching on the filesystem/-set where VM and Databases-files reside, but that counts for ZFS as well...well for all CoW-FS's in fact.

Can you explain the reasoning for this?

On a copy-on-write filesystem, modification of part of a file means fragmenting it, which carries a large performance penalty. Disabling the CoW behavior means you either cannot take snapshots of the file, or that snapshots need to duplicate the file in its entirety, but it allows for quick modification of the file in-place without fragmenting.

Is that assuming HDDs not SSDs ?

I understand disabling CoW for databases, as the DB itself will manage it, but why disable caching?

Depends on the RDBMS, but for big ones it's common practice to tune[0] them to allocate a big chunk of memory (or all of it) and let them manage caching by themselves. After all, they know better about the structure of the db files than the OS.

[0]https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serv... Relevant parameters: effective_cache_size/shared_buffers/work_mem

Oracle offers a document for running their database on ZFS.

https://www.oracle.com/technetwork/server-storage/solaris10/...

"The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files."

"Starting with Oracle Solaris 11.1, blocks that are referenced from either the snapshot or from the database image actually share ZFS cache buffers for a reduced memory footprint."

"Taking a snapshot of a database can involve placing the database in hot-backup mode just for the instant necessary to capture the ZFS snapshot. The database can then be returned to normal operation."

"A clone is a full read/write accessible fork of the original database. Therefore, by using either the primary or a replicated storage system, you can cheaply create hundreds of copies of your up-to-date database to run test and development workloads under real data conditions."

They also allow compression of the archived (intent) logs.

In case of Postgres it knows better what to cache, you can let zfs additionally cache but it's a bit of waste, better give that ram to postgres than to Arc.

I think PostgreSQL actually is a bit of an odd one in that it relies on the OS to do proper caching of whatever data is read from disk. It has its own buffers for various things, but I'm pretty sure it doesn't (yet) implement its own disk cache.

I don't think CoW can be disabled on ZFS... which is fine because VMs and databases pretty well with ZFS, even with CoW (a block size adjustment might be needed, though).

Yes true, but with Databases you should change the recordsize to 8 or 16k (Mysql or Postgres) and tune full_page_writes in postgres to off (zfs takes care of that), and just cache the metadata because the postgres-cache is better informed of what to cache.

How does ZFS avoid the performance pitfalls of btrfs's CoW implementation for VMs and databases?

No, with ZFS you just tune the recordsize of the dataset to match your database record size.

> I wondered mainly how they'd handle the various cases where btrfs does not work well, e.g. files that change often inline (databases, VMs, etc).

It least for default libvirt locations fedora disables CoW (chattr +C). No change in software needed.

This is what I was thinking as well. It just means packagers have to be mindful of what kind of files their maintained software makes and to appropriately carry these metadata changes where they are needed. I don't really see an issue with this mindset.

Reply on news.ycombinator.com