Hacker News Reader

Comment by the_hoser | Hacker News Reader

The degree of hold-my-beer here is off the charts.

It's not quite as dangerous as you'd think.

The standard technique is to reserve a big file on the old filesystem for the new filesystem metadata, and then walk all files on the old filesystem and use fiemap() to create new extents that point to the existing data - only writing to the space you reserved.

You only overwrite the superblock at the very end, and you can verify that the old and new filesystems have the same contents before you do.

I believe that is also the method [btrfs-convert](https://btrfs.readthedocs.io/en/latest/Convert.html) uses. A cool trick that tool uses is to keep the ext4 structures on disk (as a subvolume), which allows reverting to ext4 if the conversion didn't go as planned (as long as you don't do anything to mess with the ext4 extents, such as defragmenting or balancing the filesystem, and you can't revert after deleting the subvolume of course).

I believe you are right. You can only convert back to the metadata from before. So any new or changed (different extents) files will be lost or corrupted.

So best to only mount ro when considering to rollback. Otherwise it's pretty risky.

No, it also covers the data. As long as you don't delete the rollback subvolume, all the original data should still be there, uncorrupted.

Even if you disable copy-on-write, as long as the rollback subvolume is there to lay claim to the old data, it's considered immutable and any modification will still have to copy it.

I understood it as "it doesn't touch/ignore the data". But I guess we mean the same thing.

You are right. All of the old files will be in areas btrfs should consider used. So it should correctly restore the state from before the migration. Thanks!

I tried that on a system in 2020 and it just corrupted my new FS. Cool idea though.

You don't understand. You did get a btrfs that worked normally. /s

My conversion went fine, but there were so many misaligned sectors and constant strange checksum errors (on files written after the conversion). With the cherry on top being that if there’s more than X% of checksum errors, btrfs refuses to mount and you have to do multiple arcane incantations to get it to clear all its errors. Real fun if you need your laptop for a high priority problem to solve.

Lesson learned: despite whatever “hard” promises a conversion tool (and its creators) make, just backup, check the backup, then format and create your new filesystem.

I've never had the conversion corrupt a filesystem for me (plenty of segfaults halfway through, though). It's a neat trick for when you want to convert a filesystem that doesn't have much on it, but I wouldn't use it for anything critical. Better to format the drive and copy files back from a backup, and you probably want that anyway if you're planning on using filesystem features like snapshots.

Windows used to feature a similar tool to transition from FAT32 to NTFS. I'd have the same reservations about that tool, though. Apple also did something like this with an even weirder conversion step (source and target filesystem didn't have the same handling for case sensitivity!) and I've only read one or two articles about people losing data because of it. It can definitely be done safely, if given enough attention, but I don't think anyone cares enough to write a conversion tool with production grade quality.

This is a weird level of pedantry induced by holding many beers tonight, but I've always thought of "Hold my beer" as in "Holy shit the sonofabitch actually pulled it off, brilliant". I think it's perfectly fitting. Jumping a riding lawnmower over a car with a beer in hand but they actually did the math first. I love it.

It’s referring to a comment a drunk person would make before doing something extremely risky. They need someone to hold the beer so it isn’t spilled during what’s coming next.

Right, but in those situations they succeed, kind of like "the cameraman never dies".

{\off I think they used "hold my beer" correctly. It can be used for any weird idea, that a drunk person would actually try (usually with a stretch), regardless if they succeed or not. I don't think that "the SOAB actually pulled it off" is part of the usage.}

A couple of years ago it was more like juggling chainsaws: https://github.com/maharmstone/ntfs2btrfs/issues/9

I tracked down a couple of nasty bugs at that time playing around with it, hopefully it's more stable now.

Apple did something like this with a billion live OS X/iOS deployments (HFS+ -> APFS). It can be done methodically at scale as other commenters point out, but obviously needs care).

When this first showed up I took 3 backups: two on networked drives and one on an external drive which was then disconnected from the system.

The second time I just went “meh” and let it run.

Craig Federighi on some podcast once said they conducted dry-runs of the process in previous iOS updates (presumably building the new APFS filesystem metadata in a file without promoting it to the superblock) and checking its integrity and submitting telemetry data to ensure success.

You can do all the testing in the world, but clicking deploy on that update must have been nerve wracking.

Apple doesn’t just deploy to the whole world in an instant though.

First it goes to the private beta users, then the public beta users, and then it slowly rolls out globally. Presumably they could slow down the roll out even more for a risky change to monitor it.

Sure, but still whoever wrote the patch had his ass on the line even shipping to a batch of beta users. Remember this is Apple not Google where the dude likely got promoted and left the team right after pressing click :)

Very much worth saying that many of the people what write code at Apple are women. Some of them are my friends.

You don’t need to look that far. Many of us here lived through the introduction of NTFS and did live migrations from FAT32 to NTFS in the days of Windows 2000 and Windows XP.

I still remember the syntax: convert C: /fs:ntfs

IIRC there was a similar conversion tool in Windows 98 for FAT16 -> FAT32.

Yea thanks for recalling this. I totally forgot about that because I never trusted Windows upgrade, let alone filesystem conversion. Always backup and clean install. It's Microsoft software after all.

Note this is not the Linux btrfs:

"WinBtrfs is a Windows driver for the next-generation Linux filesystem Btrfs. A reimplementation from scratch, it contains no code from the Linux kernel, and should work on any version from Windows XP onwards. It is also included as part of the free operating system ReactOS."

This is from the ntfs2btrfs maintainer's page.

https://github.com/maharmstone/btrfs

It's the same file system, with two different drivers for two different operating systems.

The metadata is adjusted for Windows in a way that is foreign to Linux.

Do Linux NTFS drivers deal with alternate streams?

"Getting and setting of Access Control Lists (ACLs), using the xattr security.NTACL"

"Alternate Data Streams (e.g. :Zone.Identifier is stored as the xattr user.Zone.Identifier)"

Not sure what point you're making here. WinBtrfs is a driver for the same btrfs filesystem that Linux uses. It's most common use case is reading the Linux partitions in Windows on machines that dual-boot both operating systems

What? Why would you need a Linux NTFS driver to read a btrfs filesystem? that makes no sense.

Storing Windows ACLs in xattrs is also pretty common (Samba does the same)

I'd delete my comment if I could at this point.

Yes it is?

As someone who has witnessed Windows explode twice from in-place upgrades I would just buy a new disk or computer and start over. I get that this is different but the time that went into that data is worth way more than a new disk. It's just not worth the risk IMO. Maybe if you don't care about the data or have good backups and wish to help shake bugs out - go for it I guess.

And the new disk is also likely to have more longevity left in it, doesn't it?

If only it had a native filesystem with snapshotting capability...

Which "it"?

Both btrfs and NTFS have snapshot capabilities.

It's so well hidden from users it might not as well exist. And you can't snapshot only a part of the ntfs filesystem.

The userland tools included with Windows are very lacking, but that's more of a distro problem than a filesystem problem. VSS works fine -- boringly, even -- and people take advantage of it all the time even if they don't know it.

Reply on news.ycombinator.com