Ext4: fix data corruption caused by unwritten and delayed extents
mail-archive.com
mail-archive.com
>So it's pretty hard to hit this bug by accident
>It requires the combination of (a) writing to a portion of a file that was not previously allocated using buffered I/O, (b) an fallocate of a region of the file which is a superset of region written in (a) before it has chance to be written to disk, (c) waiting for the file data in (a) to be written out to disk (either via fsync or via the writeback daemons), and then (d) before the extent status cache gets pushed out of memory, another random write to a portion of the file covered by (a) -- in which case that specific portion of (a) could be replaced by all zeros.
I also had multiple corruptions of files written on ext4 patition but the /boot/efi corruption rules out ext4 bug.
EDIT: It was with 4.0.2 kernel on Debian
Ted Tso about ext4:
> P.S. It's bugs like these which is why I'm always amused by people
> who think that just because a file system is safely being used by
> their developers, that it's safe to throw production workloads on
> them.
So there.
1) Support for large volumes: 8EB vs 16TB (Ext4)
2) Support for large file sizes: 1EB vs 16TB (Ext4)
3) Support for large number of subfolders: Ext3 has a 32k subfolder limit
4) Online defrag support (supported in Ext4)
For us, we had to use XFS for MySQL multi-tenent applications.
* XFS had corruption issues back in the day with specific kernels (2.6.9, that's the one I remember).
df -h | grep md127 ; mount | grep md127
/dev/md127 26T 14T 12T 55% /home/data
/dev/md127 on /home/data type ext4 (rw,relatime,stripe=1792,data=ordered)
Ext3 was 32k, but ext4 is 65k subfolder limit. I have run into this one before. The real solution is to do directory hashing when you get this crazy with tons of subfolders.
"EXT4 [on RHEL7] [Cerfified max FS size:] 50TB [Theoretical max FS size:] (1EB)"
Moreover, RedHat's lack of Enterprise Support for a configuration doesn't mean that that configuration is a bad idea, or is doomed to failure. :)
So, firstly, both Red Hat and ext4 support FSs larger than 16TB. So, yeah.
Secondly, I mean, Ted Ts'o works for Google, rather than Red Hat, so it's not like Red Hat has any special insight into the inner workings of ext4. RHEL's lack of "support" for ext4 FSs large than a certain size wouldn't make me wary of using FSs larger than that size.
Ts'o & co. seem to be pretty careful, so I would expect that the worst that would happen would be performance issues.
Regarding Tso & co, I am not so optimistic. How they communicate on the mailing list is more like: "Look we wrote this, it runs on my laptop fine it might work for other workloads too". There is no extensive test coverage run by against ext4 (at least I am not aware of it). This is not the first data loss bug in the ext4 codebase. Few years from now when enough users start to use it an get rid off these serious bugs from the code, I might even consider POCing it again.
I read that mailing list tone as something more like "This software is offered without warranty and might shave your dog and weld your toilet seat down.". I mean, folks on the LKML talk in that manner about their patches, too.
Also, even XFS has had severe data loss bugs in the distant past [1] and much more recently. [2]
[0] http://oss.sgi.com/cgi-bin/gitweb.cgi?p=xfs/cmds/xfstests.gi...
[1] https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.1... (Note in particular the last comment that describes this as "not a bug, but a feature!".)
[2] http://thread.gmane.org/gmane.comp.file-systems.xfs.general/... (fixed in RHEL in 2013(!))
But in the age of ZFS, filesystems like XFS and JFS have proven themselves worthy to remain in the ring as non-versioning, simple filesystems. Their resilience (now, despite any known issues in the past) is beyond reproach, combined with their speed and performance (comparing apples to apples, not journaled to non-journaled) is where it should be.
Why does any OS/distro use extfs by default for new installations? I can understand ancient machines that upgrade from ext3 to ext4 to avoid migrating data to a new filesysystem/mountpoint, but for new installations?
How is resilience characterized for filesystems which are not widely adopted and thus exposed to diverse usage scenarios?
But XFS and JFS have had their share of corruption bugs too. I don't believe any were long-standing bugs, though.
Wikipedia mentions that JFS has a misfeature where write can be delayed so long (indefinitely) that a power outage can cause major data loss.
Didn't Suse ship with JFS as default out of the box for a very short period of time?
Nothing compares to having decades of experience with an on-disk format, a fsck tool, etc. Ext3/4 has had the most users, therefore it will be the most trusted (bugs like this one notwithstanding). When you're choosing the distro default, you have to prefer reliability to performance/features.
Hm, I guess I just gave an argument for why btrfs won't become the default for another 5-10 years. We'll see!
There are long standing bugs related to ENOSPC and new ones still regularly pop up, often related to the (fixed) separation of metadata vs data allocation. In general, disk space usage handling is awkward and integrates horribly with existing tools (df/du), especially when snapshots are involved.
Snapshot/subtree handling and deciding what gets automatically mounted in parent/child relations is pretty weird and the whole subsystem probably needs a rewrite.
Online dedup.
They probably also want to fix the whole extent design that causes random-access files to take up to N*(N-1)/2 disk space, where N is the original file size.
What's wrong with how compression works? From a user's perspective, you either set a mount option, set a bit on a file attribute, or explicitly call for compression with btrfs defrag. [0] If you combine the latter two operations into a single checkbox, this is exactly how NTFS handles things. What am I missing here? Also, I can't agree that modern[1] btrfs handles unexpected power cuts poorly. This just hasn't been my experience. I can, however, agree that log replay isn't complete and still needs work.
I can't speak to the ENOSPC bugs, I haven't run into any in a very, very long time. Some time after 3.14, btrfs got a pool of space called the "Global Reserve" [2] which was intended to address ENOSPC issues. Somewhere around that time, btrfs also grew a better btrfs-specific df function invoked with "btrfs filesystem usage". I don't make extensive use of snapshots, but the numbers I get out of btrfs fi usage almost exactly match the numbers I get from plain old df.
I'm unware of snapshot automounting. Can you help me understand what this is? (A brief Google search wasn't enlightening.) I agree that taking away the ability to have subvolumes outside the FS tree if you didn't set things up just right at FS creation time is pretty bullshit. That certainly needs reworked. Everything else in my limited experience with subvol management seems okay to me. What's seems strange to you?
Btrfs doesn't yet have built-in online dedup, but are you aware of dupremove? [3]
Can you give me a recipe to create an N(N-1)/2 sized random access file? I use BTRFS for some very small random-access databases and large pretty-much-append-only databases and haven't run into this behavior.
I have a couple of things to say about your other comment:
People generally don't write to a mailing list to say what a good time they're having with your software.
It's pretty shit that log replay isn't worked out yet.
I certainly* don't intend to stop making backups. OTOH, I've been "getting lucky" for five years straight while running a FS that many people regard as the most untrustworthy thing in the world on a drive that many people regard as entirely unreliable and untrustworthy. :)
Cheers!
[0] Conceptually, this makes a lot of sense, 'cause all defrag does is re-write the file. NTFS re-writes files when you request that they be compressed, too.
[1] "Modern" means btrfs from the past two years or so.
[2] Much like ext*'s reserved-for-the-superuser space.
Now... if you're using Kubuntu 15.04[3], with btrfs as your root volume, know that system lockups that require a hard reset may well do something to the log that makes mount hang. Do this [4] if your "task is hung" backtrace looks something like the one in the wiki page, then btrfs-zero-log is the thing you need to do.
[0] The drive is still going strong and error-free, too! :D
[1] This drive is what taught me to ALWAYS upgrade the firmware in your SSDs, ASAP. :P
[3] Christ, what a nightmare this version is! :(
You have been extremely lucky. Keep making backups, though.
For most uses, no-one will ever spot the difference between an underlying modern(-ish) filesystem. Ext3/4 also has a very decent set of maintenance / recovery tools and is well supported by most distributions. That is plenty good enough for almost everybody.
First, people like boring in filesystems, and ext4 was seen as a natural and less risky successor to ext3, which most people ran. And I think that's a reasonable perception.
Second, ext4 went out of its way to solve some of the system-level issues that tended to cause problems, before it went into production. For instance, the ext4 developers introduced hacks to deal with software that handles write atomicity incorrectly, making it much less likely that you'd end up with a zero-length file if you cut power or crashed at the wrong time. XFS and JFS had similar issues that actually hit users, breaking invalid-but-widespread assumptions about filesystem semantics. ext4 at first considered having similar semantics, but instead worked around how many programs actually wrote files, which made it safer in practice.
(Also, "in the age of ZFS"? ZFS is by no means the dominant filesystem, or even an obvious contender for that title; it has a few popular features, but it's hamstrung by its choice of license.)
Source: most benchmarks involving Linux filesystems
I assume/hope that if this was a common occurrence it would have been found much earlier...
Update: Having reread the report, it appears that this may only be a problem in the new 4.0 kernel?
Update2: Nope, not just 4.0 kernels (see below).
I see no indication that this is a 4.0-specific issue. This code has existed for a long time. They fixed it in "stable kernels 4.0.3 as well as much older stable kernels."
edit: major corruption is possible, not just slight, because an extent can be 128Mib on a 4K block size filesystem.
No, reporter uses 4.0 kernel which is why bug report lists that version. I've looked at commit message for the fix but it doesn't talk about any regression, so this bug may be existing for a long time.
Ok, so I think what's happening here is
1) Write data
2) Busy filesystem or something makes the write of the
extent delayed
3) An update to that data attempts to get written
4) The delayed extent gets replaced with new data, but then marked as written here
instead of when it hits the platters
5) Now data loss has happened because there are no
delayed buffers
http://www.spinics.net/lists/linux-ext4/msg47782.html
edit: this type of problem wouldn't affect a COW filesystem because when you overwrite existing data you aren't actually overwriting it, you're writing new data and updating the map of where to find the entire file contents.
I think HN submissions should have some standards for stories that will create the appearance of "there is a critical problem with software X that lots of people use, and everyone should panic and upgrade" announcements. Having the authors of software X involved in the announcement seems like it would be a good place to start.
(I'm not saying that Josh is mistaken about having been hit by this bug, or was involved in the decision to submit this to HN.)
The project maintainers are in the best position to make that call.
"A single extent in ext4 can map up to 128 MiB of contiguous space with a 4 KiB block size."
HN has such a standard: the guidelines ask for titles not to be misleading or linkbait. (Not an opinion on the current bug.)
If I interpreted the commit message correctly, this bug is not a new thing at all and it has been there for years, but it has been noticed only lately, which means we're no better applying the fix now than waiting until distribution packages ship it.
I was not involved, and it's sounding like this is a different bug than the one I got hit by. This bug still qualifies as "critical" in a Debian bug-reporting severity sense ("causes serious data loss"). I don't know what the original title was though.
ouch