An implementation of the NTFS filesystem in a Rust crate | Hacker News Reader

An implementation of the NTFS filesystem in a Rust crate

github.com

127 pointsolvy053 comments

Question for the experts!

Has NTFS changed much in... 20 years?

Linux filesystems seem to be evolving all the time (try to install any new distro today and check out how many filesystems you can pick from the dropdown...)

Apple replaced one filesystem with another in just the past few years, with apparently some very advanced new features.

But NTFS is just... NTFS? Does NTFS today look anything like how it looked in 1994? Or 2018? Does it change radically between versions of Windows? Or does the MS creed of compatibility at all costs mean it's basically a filesystem unchanged from the 90s? Thanks!

> Linux filesystems seem to be evolving all the time (try to install any new distro today and check out how many filesystems you can pick from the dropdown...)

This sounds like evolution of options more than evolution of existing file systems

Is ext4 still not most popular for most end users? Curious how much ext4 has changed vs NTFS past... ~20 years (ext4 went stable 2008 apparently)

ext4 remains the default for a lot of distros and probably most end users, but other options are common as well. openSUSE and Fedora now default to btrfs, RHEL defaults to XFS.

XFS also has not changed much in the past 10-20 years :-).

https://en.wikipedia.org/wiki/XFS lists what I would call significant improvements, in particular,

> Linux kernel 4.8 in August 2016 added a new feature, "reverse mapping". This is the foundation for a large set of planned features: snapshots, copy-on-write (COW) data, data deduplication, reflink copies, online data and metadata scrubbing, highly accurate reporting of data loss or bad sectors, and significantly improved reconstruction of damaged or corrupted filesystems. This work required changes to XFS's on-disk format.

Wow you've missed so much. It's gone from "only fast with huge files" to being one of the most performant and capable filesystems in existence

My bet is that btrfs will replace ext4 in the nearish future. It's the default for Arch, which also comes default with filesystem snapshots before any system updates.

It's pretty nice. I think it also supports some flavor of volumes or virtual partitions, but I don't have any real use for that kind of thing.

The snapshots are really good though. Very fast and don't take up a ton of space, somehow.

Never using btrfs no matter how default it comes. Only filesystem I've had data loss with, 2 district occasions.

Would you mind noting how recent the data loss was?

Latest in the past year or so. All long after it was "stable".

Appreciate it. As someone who basically continues to use ZFS happily and didn't kick the tires because of the stability & ergonomics issues in the past, this is concerning to hear...

Looks like kicking the tires will remain somewhat in the future for me.

It's much stable after 6.4 kernel. Last year was not really stable yet. I faced too, problem is I didn't know we have to schedule a `btrfs scrub`. After kernel update and scrub, everything works smooth.

I recently had a bit of a data hiccup on my desktop laptop, where I'm using Btrfs with LVM encryption. What's weird is that I started getting these random reboots out of the blue, with no error messages to explain what was going on. I was scratching my head for a while, thinking it might be a hardware issue.

Then, in frustration, I rebooted into my Windows partition, which I rarely ever use. to see if I could back up my stuff and figure out why I wasn't having the same reboot problems in Windows.

After trying a bunch of Btrfs repair commands over a bootable usb disk I found online, I hit a wall with some errors that couldn't be fixed. So, I threw in the towel and reinstalled my system with ext4. It's been running smoothly for three weeks now with no more error headaches.

Could not find the logs of the errors anymore unfortunately to report.

Me too, also on two distinct occasions. It kept going read only on a third distinct occasion and i copied my stuff off and switched to zfs and haven't looked back. I am hoping bcachefs provides a compelling option in the next decade to switch off zfs but its a tall hill to climb

yeah, I managed to corrupt my btrfs on my laptop a few months ago (loose memory stick causing crash at bad time, apparently). I was able to recover almost everything with btrfs-restore (my latest backup was several weeks old then) but it was still not a pleasant experience...

Same, I can reliably break my NVMe RAID10 by simply pulling power

Not like, "hehe we need to sync", but "good luck repairing, enjoy probably formatting"

LVM/MD RAID, ZFS, etc all behave fine. Only BTRFS is this fragile.

Btrfs seems to be the new standard, it might well end up being the one big thing Oracle did for Linux; it definitely evolved significantly over the last 15 years.

Ext4 has changed their binary format twice with only forward compatibility. Mounting a new, large ext4 partition often won't work on old machines.

Source: had to patch yaboot(PowerPC Mac bootloader) to update ext4 to boot with /boot on ext4.

ext2/3/4 are the same on-disk filesystem with features added, so could also compare the timeline from the 90's. NTFS and ext2 have the same year (1993) listed on WP as intro date.

NTFS has gained support for a few features over the years, but it wasn't meant to evolve, much. Microsoft knew what they wanted from the filesystem and they wrote it to spec very early in its life.

ReFS on Windows is where the changes are coming to. finally a 1st party filesystem which supports Copy on Write file cloning. a lot of core features of NTFS aren't supported on ReFS, so it's clearly not intended to be a replacement for NTFS.

Windows recently got support for booting from ReFS volumes, so it seems like Microsoft has started working on it again and maybe does think a move away from NTFS is possible in the future.

until ReFS supports compression, encryption, quotas, OneDrive support, and whatever else is missing, NTFS isn't going anywhere.

they're gonna live side by side for 30+ years. they are meant for different uses.

The recent interview with Dave Cutler mentioned this briefly. When the Cairo team was working on their object file system, they had no requirement to maintain backwards compatibility (which makes sense for new development). But they didn't build in a way to have in-place upgrades after it would have been released, either.

The Tukwila team, still working with the old codebase, had a rule that NTFS could be upgraded, but it had to either accept the old file system structures, or be upgraded in-place. And their code allowed for that. Relevant portion starts at about 2:10

https://www.youtube.com/watch?v=xsQBKkPj5Nc

There’s ReFS for windows. Don’t forget Apple was stuck with one barely functioning FS (HFS+) for decades.

Is ReFS real? Like does anyone really use it outside of Microsoft's labs?

The Windows 11 feature "Dev Drive" uses ReFS.

https://learn.microsoft.com/en-us/windows/dev-drive/

Pretty cool, I'd never seen that before. Thanks

It hasn't changed radically, but it has changed. Wikipedia actually has a nice write-up if you want the details, but support for symbolic links, transactions, and partition resizing are all things that have materially improved my life in the last ~5-10 years.

All of those features are over 10 years old. Time moves fast. Windows 7 was 14 years ago.

> Windows 7 was 14 years ago.

Surely you lie..

And here I am, still trying to rid the corporation from Office 2010 from some users..

Unfortunately(?), Transactional NTFS is deprecated and Microsoft claims it may not be available in future versions of Windows.

linux ntfs support seems sort of hit or miss.

For example, I have a dual-boot arch/win system with 2 ntfs partitions and a with a recent update the linux kernel ntfs driver can't mount the filesystems, have to fall back to the fuse implementation.

It also didn't seem to be able to do things like set system flags hsra, but I found a way with the fuse implementation.

This is pretty great. I've been wanting to make a simple utility that can scan an NTFS drive for a filename pattern; scanning the MFT is incredibly fast versus using the actual filesystem API calls to do so.

Sounds exactly like Everything (https://www.voidtools.com/) - do check it out if you're not aware of this tool.

Everything is one of those things that are so obvious that it hurts it's not offered by default on operating systems (even Linux and Mac).

Why go the effort to index the files in a filesystem when the filesystem IS an index in itself? Just query it directly

The locate or mlocate command does that on Mac/Linux.

Looks like it's read-only at the moment, but still pretty cool.

It's dual-licensed as MIT or Apache 2.0, so it ought to be usable just about anywhere.

> Looks like it's read-only at the moment, but still pretty cool.

That's how Linux NTFS support started out, too.

What is it about NTFS that makes writing hugely more difficult than reading?

It’s probably mostly done out of carefulness. I’d do the same if I were to write my own implementation of a file system.

Bugs and mistakes in the code are already not great when implementing the read part. But if you mess up the writing well now you’re gonna cause corruption of existing data. No one likes that :)

Better to stick to just reading for a long while, and weed out most of the bugs and getting an even deeper understanding of the filesystem, before starting the implementation of anything that will modify the data.

> I’d do the same if I were to write my own implementation of a file system.

Exactly. I have my own ext2 driver for a couple of hobby OS projects where the read side is extremely stable, but as the write side involves inode allocation, block/bitmap allocation etc. I don't turn it on on outside of testing since I know it's going to destroy the disk images until I "get it right".

I would expect writing to be much harder on all file-systems. There's only one way to read the data, but there could be any number of strategies for deciding where to write new data.

Reading is always easier to implement than writing. Only writing has the potential to actively corrupt the filesystem or leave it in some inconsistent state.

Generally, the more modern the filesystem, the more complicated writing becomes.

That's the Rust standard—Apache 2.0 for its patent license, and MIT for its GPL-2.0-only compatibility.

Author here. Surprised to suddenly find my project from 2021 on Hacker News!

If you want to learn more about NTFS internals and my crate, check out the talk I gave at FOSDEM 2022: https://archive.fosdem.org/2022/schedule/event/misc_ntfs_rus...

Also I'm open to any questions here.

Before you get too excited it is read only.

Probably prudent - would not trust any implementation except native ones in Windows that does writes.

well, you should. write support for NTFS is supported by quite a few tools, now, including the Linux Kernel.

And I've had the new linux NTFS driver corrupt files more than once.

In fact, it just corrupted the partition table of my external backup drive this year. Which was last time I trusted it for anything but read-only mounting.

and on the flip side I have been doing r/w to NTFS from linux for yeeeeears with never an issue.

And I have had /windows/ corrupt NTFS volumes.

Personal anecdotes rock :D

I've used it extensively on multiple machines and have had no problems. ¯ \ _ ( ツ ) _ / ¯

Why is this even an answer? ¯ \ _ ( ツ ) _ / ¯

It have corrupted my drive several times.

Reply on news.ycombinator.com