Hacker News Reader

Comment by otterley | Hacker News Reader

NTFS is in some ways arguably still more advanced than ZFS: think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

Also, NTFS already has snapshots via Volume Shadow Copy.

ZFS is great but it’s not going to be a viable replacement as it exists today.

View on HN

> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

NTFS's USN change journal (which is separate from the transaction journal) is also useful as a robust offline alternative to ReadDirectoryChangesW file monitoring. Toy example: https://gist.github.com/pervognsen/bcc610d6a5ae6cbc3b2b4f7aa....

Anyway, the problem I've always had when using non-standard file system metadata like alternate streams or sparse files is that they interoperate poorly in practice. There's lots of code and programs out there that wants to treat everything like a plain old file and hence won't preserve alternate streams or sparseness. From my recollections this is true even for several of Windows's built-in utilities. Same issue if you want to transfer files over a socket or pipe or a non-NTFS storage medium like a FAT-formatted USB drive or a NetApp filer or a source control system. As a result this kind of thing is only really useful when deployed in a bubble; I believe WSL uses alternate streams for Unix permission bits and other Unix-specific file properties. Even though those WSL guest files live in a directory tree on a normal NTFS volume, they strongly caution you against touching them directly outside of WSL, presumably for the aforementioned reasons.

Aside from features, NTFS has some performance issues compared to other modern file systems, notably with lots of smaller files.

Some minor notes:

- For permissions WSL uses EAs; for capabilities it uses ADS

- I'm not sure if it's really NTFS that is slow with lots of small files or the I/O subsystem in general... I got the impression it's the latter but not sure.

Not a lot of this matters in the real world. The real shame is the shitty small file performance due to MFT contention on NTFS.

It absolutely matters: a fair amount of software, both enterprise and consumer, already uses these features. Malware detection, for example, makes heavy use of reparse points.

Windows Update and Windows Installer use NTFS transactions.

From Microsoft's documentation:

Microsoft strongly recommends developers utilize alternative means to achieve your application s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows. For more information, and alternatives to TxF, please see Alternatives to using Transactional NTFS.

https://docs.microsoft.com/en-us/windows/desktop/fileio/abou...

I'm pretty convinced that documentation is BS. I could be wrong I guess, but I honestly don't see them removing TxF anytime soon. They didn't exactly build their business on breaking backwards-compatibility of APIs.

They've burned a lot of APIs in the last few years.

Examples?

https://docs.microsoft.com/en-us/windows/desktop/apiindex/wi...

They're not removed though? They're 'deprecated' as in they don't want you to depend on them, but they're still very much there and usable.

ZFS snapshots work well enough for that.

The Sun Storage Appliance used ZFS snapshots for updates, and it worked quite well.

Did that change recently? I haven't used Windows since XP, but back then, they used Volume Shadow Copies (VSS) for making backups used for reverting changes, not NTFS transactions.

Windows Update switched to using NTFS transactions in Windows Vista.

Reparse points could be emulated at API level.

Is the MFT even documented and available via APIs on Windows? Could NTFS (at the expense of losing backwards compatibility in the file system (which already only extends to features that have been there before)) just change how the MFT works transparently for well-behaved applications (i.e. those that do not grovel around in undocumented data structures)? Admittedly, there may still be a number of applications that decide to »optimize« a system by digging around in parts that they have reverse engineered, and those might simply corrupt the file system when running on a newer NTFS version, but I guess that problem may have existed previously as well.

MFT is an internal structure. They could fix it TBH but the issue is that this would require an NTFS filesystem revision and for those of us who suffered through NT4's broken ass filesystems might start complaining :)

> think reparse points, alternate streams, transactions, BitLocker integration, and shrinkable volumes, to name a few.

All of these except shrinkable volumes (somewhat) are already part of ZFS, also assuming that you replace "BitLocker integration" with more general "encryption integration"

ZFS and NFSv4 were both designed to be able to serve the entire subset of NTFS features over the network, to interop with Windows computers.

Reparse points and ADS? Do you have a reference handy? I’ve never seen such things or anything like them in ZFS.

Reparse points are basically more primitive directory-only symlinks, aren't they? Even the symlinks introduced in Vista are better...

As for ADS: basically xattrs. They can be arbitrarily named and each xattr on ZFS can be up to 16EiB large (same as the primary bin for file content). (As an aside, they aren't fully usable on Linux since Linux itself imposes a 64KiB limit on an xattr's content -- but that's not a ZFS limitation)

Reparse points are directory entries that are special and interpreted by the file system driver (or some other higher-level component that's not the file system itself). Directory junctions (which you mean here, I guess) are reparse points, as are symlinks and a bunch of other things.

> Reparse points are basically more primitive directory-only symlinks, aren't they?

Absolutely not! Reparse points are directories that are associated with drivers that extend the capability of the filesystem itself. They can be used to implement symlink-like behavior, but that's only a single use case that scratches the surface of their potential power.

See https://docs.microsoft.com/en-us/windows/desktop/fileio/repa... for more information.

Thanks, I wasn't aware of that... terms seem to get interchanged in the Windows world often enough.

It doesn't sound like it is really an inherent property of the file system, but just controlled by a driver. I don't see why they couldn't be made to work on any other file system, such as FAT or ZFS.

Though it might finally do away with the ever-persisting Windows error messages about "this file is already open" followed by a search through all of the processes that might have it open to terminate the offender.

That error is orthogonal to the filesystem in use. Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.

> Every OS that provides exclusive locking, regardless of filesystem, will return an error if more than one process wants to acquire such a lock.

Obviously.

The super annoying thing is when the OS doesn't try to tell you which process(es) keeps it open and doesn't even ship with built in tooling to let you find out on your own.

I think that is the point GP tries to make.

Open up "Resource Monitor" and go to the "Disk" tab. Under "Disk Activity" you can see all open file handles.

Which are thousands...

You need the Sysinternals utilities to make life tolerable: https://docs.microsoft.com/en-us/sysinternals/downloads/hand... from the command line or Ctrl-F in https://docs.microsoft.com/en-us/sysinternals/downloads/proc... if you prefer a GUI. They do a brute-force search over the open handles in all processes, but it's effective.

I prefer this utility: http://www.emptyloop.com/unlocker/

It lets you see which process locked a file, and optionally remove the lock.

Certainly not a sensible solution like having a dialog button that gives you all the relevant information immediately and empowers you to solve the problem without a goose chase.

    PS> $lockedFile="C:\Windows\System32\wshtcpip.dll"
    PS> Get-Process | foreach{$processVar = $_;$_.Modules | foreach{if($_.FileName -eq $lockedFile){$processVar.Name + " PID:" + $processVar.id}}}

That foreach looks like it should really be a where

It was copied from stackoverflow... apologies, I just knew it was possible.

    PS C:\> gps |? {$_.Modules.FileName -match "wshtcpip\.dll"} | select name, id

The way filesystem locking works on Windows is an intentional decision and not an NTFS thing. It's there for a good reason even if it's inconvenient. The alternative has some real downsides. "Two CMD windows have the same CWD but are showing different folders" is not a user-friendly experience.

I bet pretty much every Windows user would trade confusing command prompts in strange circumstances for not having to reboot for pretty much every single Windows Update (due to it being unable to update files that are in use)

It would be horrendous to even try to troubleshoot a system with a dozen different version of OS DLLs loaded because the system had not been rebooted for a dozen patches.

Or, imagine every copy of Word you have running is using a different set of binaries.

No, thank you.

And yet Linux distros some how manage years of uptime with continuous updates.

Send To > Unlocker

Hasn't worked since XP, has it? I'd love to see the source for that - I don't know how they were hooking the Copy dialog.

http://www.emptyloop.com/unlocker/

I use it in Win 7; I think it works most of the time (not always).

download sysinternals tools, one of which is "handle" for quickly finding open files.

Yeah, it should come with the OS, but that's Windows for you.

If a file is damaged by bit rot, NTFS will happily serve up garbage. So those are nice features, but viewed in that light they're little more than window dressing.

You say that as if bit rot is the black plague of data storage. In the past 20 years I have encountered 'bit rot' with NTFS... not once. I have encountered bad blocks and replaced drives once in a while by monitoring SMART status, which probably prevented the rot. It is definitely great to have protection against it, but to call other features window dressing when others have used them more often than they thought of checksumming feature is not objectively fair.

In the past 20 years I've lost multiple files to bit rot on NTFS. In my opinion, the primary purpose of a filesystem is to reliably store data. Checksumming is not particularly complex technically, does not impose significant run time costs, and allows the filesystem to at least detect when data has been damaged. Unless there's some other very compelling reason to not do checksumming, it's reasonable to consider it's omission to be a de facto bug.

Does this mean ext4 is bugged too? We many filesystems with this bug, and we lived with this bug for a while. Anyway, I don't disagree on the technical cost/benefit analysis, but still, unfair.

Didn’t know any of that. Thanks!

Reply on news.ycombinator.com