Scripts for Btrfs Maintenance
github.com
github.com
Unbalanced drives can affect performance as one drive will have to do more of the work often bottlenecking. But except for extreme cases it is probably a rounding error.
One of the effects of balancing the tree is to release unused space from the allocation pools to general availability, solving that problem.
I thought trim was enabled by default (https://wiki.archlinux.org/title/Btrfs#SSD_TRIM). Does fstrim do something more than that or is it the same thing?
It very well may be! Depending on your distribution... they may already have similar services/timers for periodic scrubbing
On Fedora they aren't provided; but are with Arch
Unless you run an array it's fairly meaningless, beyond being an administrative-reporting tool. It can't fix failed checksums without mirrors or parity
I think the checksums are validated on read anyway, so it's mostly to mitigate bitrot - very specific to 'glacial' storage
It would, if you were running a well-designed/maintained operating system. Unfortunately many Linux distributions make poor decisions.
That will read all of the data from the storage device(s) checking for coherency. In the case of redundancy (ie: RAID1) and corruption/rot, would use the other copy to make things whole
Beyond that and trim, most of what's in here is fairly specific. ie: avoiding ENOSPC or dealing with array changes
edit: I think running this more than ~monthly is overzealous... and only really meaningful if you have redundancy
If memory serves, the checksums are validated when you read things anyway - so I question doing passes too aggressively. I'll accept a bit for bitrot
Both SSD's and spinning metal drives have ECC bits stored with the data, and the drive can detect when the data was read easily or required lots of error correction applied. And then based on that they can make the optimal call of how often to read data to see it it is 'nearly rotted' and needs a rewrite.
The filesystem has no knowledge of any of that, so has to do dumb periodic scans.
At some point after I restarted the computer and it wouldn't mount the partition saying that the path for mounting was already being used (even after restarting??), but the disk was not mounted. Searching around apparently there is some kind of bug where Linux will "cache" the BTRFS UUID [1].
At the time, the "solution" I read was to change the UUID of the BTRFS partition running btrfstune. Which I did... and it supposedly changed successfully. Except that after trying to mount the partition again, it will tell me that there was an error in the partition: the tree had mismatching UUIDs :-/. I tried updating the UUID several times, and the command ended in success, but the error remained. My BTRFS disk was officially broken...
After spending several hours trying to fix the issue following the BTRFS documentation (pretty shitty TBH), in the end, I ended up having to do `btrfs restore -iv ...` to extract the data from the disk into some other external disk (formatted NTFS this time!!). The command is still going, about 2 weeks later and I am slowly recovering my 5TB of data.
But the one thing clear to me is that I don't trust BTRFS or it's admin commands AT ALL after this experience. Once I finish recovering my data, I'll nuke the partition, format the disk in NTFS and forget about this sour experience.
[1] https://unix.stackexchange.com/questions/603528/why-am-i-get...
It doesn't even get much better with many drives and avoiding the docs/tooling
BTRFS arrays will consistently corrupt with my reset button -- RAID10 on gen4 NVMe drives... while LVM/dm-raid + traditional file systems are absolutely fine
YES!!! Sorry to beat a dead horse, but it has been so frustrating for me. I thought I did something wrong, like, how could the partition break for doing nothing! Where did I fucked up? I cannot imagine the experience with a RAIDed BTRFS array shudders
Sure... HDDs shouldn't do that, but it's literally the job of a BTRFS raid to protect me from such issues. The data was all still intact on the other drive. I could see it with dd, but BTRFS refused to read it.
[1]: https://unix.stackexchange.com/questions/634050/in-what-ways...
In theory perhaps, but absolutely not in practice. Making things atomic is tricky.
However, btrfs is a very fragile filesystem and any inconsistencies caused by the drive bugs tend to lead to it breaking and refusing to mount, whereas on other filesystems you'll just get a warning in dmesg.
It doesn't help that btrfs developers refuse to work around drive bugs.
The entire selling point of the filesystem is that it's resilient to data corruption. Drive bugs are just another class of data corruption.
I know you don't want to get asked this, and I wouldn't if you'd just said "I tried this once a while ago and it happened", but if it's consistent... have you filed a bug?
It's occurred to me, but I expect a fair bit of push back; 'you abuse the array and expect it to... what?'
I'm not really interested in debating that, and it's remarkably easy to get hung up on. I've floated this issue a few times unofficially and it's a constant stickler
I know expecting coherency in this situation is a little silly, but BTRFS is notably less reliable/robust than the alternatives
All in all... I'm avoiding a situation that may or may not occur, by not participating.
Not saying it's a good thing, but it's easier for me to just use What Works
I'd be kind of annoyed if I went to the trouble of reporting a bug and got that response. But I'd also not assume it would end that way...
The DRAM caches are pretty hefty compared to metadata, so I wouldn't be surprised - though that's about my extent of understanding.
Even then, though - other filesystems are certainly more durable, dealing with whatever these drives are doing.
FWIW, this was with four Sabrent Rocket 4.0 Plus drives in RAID10 with 4K sectors. I'm not sure if these are particularly well known one way or another for deception/trickery
They hold up just fine with either dm-raid or native LVM raid10 under more traditional filesystems (I tested EXT4/XFS, using the latter consistently)
It's not a bug. It's just a fact that every BTRFS filesystem visible to the kernel (mounted or not) should have a unique ID. Otherwise operations on one filesystem (including mounting the partition that contains it) can end up applying to the other filesystem.
>At the time, the "solution" I read was to change the UUID of the BTRFS partition running btrfstune.
The solution would've been to find out why you have two partitions with the same filesystem ID and remove one of them. One example is that if you have the BTRFS filesystem on an LVM partition and then you clone the LVM partition, you end up with two BTRFS filesystems with the same ID. I accidentally encountered it while converting an unencrypted BTRFS parition into an encrypted one by dd'ing it into a new LUKS device - when I tried to mount the encrypted partition it ended mounting the unencrypted one.
Since you say it's an external USB drive, perhaps it disconnected and reconnected uncleanly and the kernel thought there were two of that drive connected at the same time.
I think the actual dynamic filesystem discovery is in user space and kernel just gets a list of devices, so it should be even easier to change that.
AFAIK the filesystem UUID, the device UUID, or both are sprinkled all around the filesystem, I suppose for reasons of identifying data belonging to the fs in case of fsck. According to my quick research device id cannot be changed. I suppose if it could be changed by some means, then btrfs rescue clear-uuid-tree could be used to fix the other uuids.
Maybe it's not a code bug, but imposing global uniqueness requirements where you can't just `cp /dev/blkdev1 myfs.img` and be able to work with the new image seems rather like a design bug.
While not a multi-device filesystem, I believe XFS has the same behaviour and you have to change the UUID (or use xfs_copy in the first place) before you can mount a cloned block device.
(I wonder how ZFS handles this?)
> Since you say it's an external USB drive, perhaps it disconnected and reconnected uncleanly and the kernel thought there were two of that drive connected at the same time.
I also used to run btrfs in btrfs-RAID10 configuration until apparently a flapping SATA link and fsck attempts were able to break the fs completely. Full system backups were great that day. I run https://kopia.io/ nowadays every three hours during day time and I've been quite happy with it.
Nowadays I run bcachefs.. Backups are still handy :).
I suppose the reason why you chose NTFS was to be able to access the data from Windows, at least in case of emergency? Because there are a lot of filesystems that are presumably more mature than NTFS is for Linux.
I thought of going EXT4 , but as you said, NTFS is more widely supported.
If it would actually affect performance, turn on autodefrag. Be aware that running a manual defrag, instead of using the mount option, will break reflinks.
[1]:https://www.phoronix.com/news/Btrfs-Discard-Tuning-Linux-6.3
btrfs is the result of Oracle trying to clone, badly, ZFS. When they bought Sun, they discontinued development on btrfs completely, as they (thought they) owned ZFS. Nobody sponsors btrfs development anymore, and its development has completely stagnated; while ZFS, under the OpenZFS project, continues to accelerate and absolutely dominates the enterprise mission critical file system sort of market.
Just use the real deal: use ZFS.
Due to the massive data loss issues with btrfs, for example, Redhat completely removed btrfs support in 8 entirely, after the preview feature in 7 bombed. They default to, and highly recommend, XFS (as do I, its a good file system).
Another good reason is simplified tooling – ZFS has a lot of special ways of doing things different from all other filesystems.
RedHat seemed to remove Btrfs because that want to move towards Stratis, not because ZFS or XFS is good enough.
None of them are perfect yet and my best hope for the future is bcachefs.
The only times I've ever heard of this not being Ubuntu pathology is people running builds of the kernel from git, and not keeping track of the extremely rare breaking changes that effect ZFS (OpenZFS's git repo for the kernel modules are updated quickly, and breaking changes are often handled ahead of time; all of this is fixed before stable kernels get released nowadays).
Doesn't SUSE still use btrfs as their default rootfs?
Looking at `git shortlog -sne --since="1 Jan, 2023" fs/btrfs/` it seems like folks at least at Suse, Oracle, and Western Digital have all made more than a dozen commits this year.
Being able to sit back and have a big bunch o'disks that I can just add and remove devices to is convenient when combined with BTFS raid1: data will be present on at least two disks, I don't care which. I don't think ZFS is able to do this. It's obviously more suited for serious enterprise use where you have plans and budgets... ;)
(Once I too have my data eaten by it I'll no doubt be very grumpy, then again that's what backups are for so...)
More seriously, I'd give ZFS a go but it's still too much of a pain in the arse to use with Linux. If even Oracle Unbreakable Linux can't integrate it properly then what hope does anyone else have?
(Talking about making me build my own kernel modules here)
Debian and SUSE both do this perfectly and automatically.
> ...
> Due to the massive data loss issues with btrfs, for example, Redhat completely removed btrfs support in 8 entirely, after the preview feature in 7 bombed. They default to, and highly recommend, XFS (as do I, its a good file system).
Synology strongly recommends btrfs in its NAS systems and promotes that over ext4, which is also supported by Synology. [1] On the consumer side, I guess this would probably be larger than ZFS/XFS usage.
If you need a well supported NAS that anyone can operate, just slap TrueNAS onto a budget NAS box and be done with it. TrueNAS uses ZFS, and actively discourages btrfs due to a history of data loss issues.
I only use it for one thing, read only compressed lower layers. It's great for that because it can be written to at setup time with standard tools, as opposed to some extra image compile step like true compressed read only FSes, but I wouldn't use it for anything that gets written in normal use.
btrfs fi defrag -r /
https://www.usenix.org/system/files/login/articles/login_sum...
https://serverfault.com/questions/733817/does-the-max-80-use...
It sped right back up again when I deleted some stuff.
I wouldn't worry about it.
On the side of ZFS, you can take one volume of a mirrorset, move it to another system and mount it in degraded mode, and copy a bunch of new files to it. When you return the updated drive to the full set and mount it, it will resilver the older drive with the newer content automatically.
That's a killer feature. With btrfs, you have to rebalance to mirror the new files, which takes a long time as every block must be rewritten between the mirrors. ZFS knows how to move just the new files in a resilver.
* (yes, DEC reference)
edit: I initially wrote deduplicates, thanks the8472, and agree with the rest of your message as well
Anyway, deduplication can be recovered after a defrag by running duperemove or similar tools. But it's a lot of added work so it's only really worth it if fragmentation is an issue in the first place.
It's also very slow to go through that on multiple snapshots.
And I think some of the metadata doesn't like to recombine?