Linux's Modern NTFS Driver Preparing a “Hidedotfiles” Option | Hacker News Reader

Linux's Modern NTFS Driver Preparing a “Hidedotfiles” Option

phoronix.com

38 pointsrbanffy55 comments

I just wish that Windows app would use the filesystem as it's meant to be rather than just pretend "dotfiles" get hidden on Windows. Just put your data in $env:APPDATA, damnit! Don't clutter my home directory with "hidden" files that aren't really hidden! Even MS tools do it, it's frustrating.

Don't get me started on the wasteland that "My Documents" has become, given 20 years of incompetent developers.

To this day, I still have a few applications that store important user configuration files via hard-coded "C:\Program Files\Publisher\AppName" path names rather than literally anywhere else.

Another bit of incompetence that really winds me up is:

software that you can specify the install location for, so it installs a update manager at that location but then always installs the actual programs under the C drive

What the heck, I've never seen this. Hey ISVs, if you see this conversation engage me to consult on how to make your installers not drive people like us up the wall! ;)

IIRC Visual Studio technically lets you install in a non-C location but 80% of the disk usage ends up on C: anyway.

Microsoft decided at some point people hard drive were plenty big so who cares? Unfortunately SSDs came along and for quite awhile space was quite tight on them!

The "Documents and Settings" for user home directories used to annoy me because its too long to type and has spaces. Thankfully the now changed it to "Users". There was a time when some old installers wouldn't parse "Program Files" correctly because of the space. Also "ProgramData" is inconsistent with the others because no space. But that folder is now hidden anyway. No consistency at Microsoft.

> There was a time when some old installers wouldn't parse "Program Files" correctly because of the space.

That's one of the reasons why it had a space. If you wanted to stop using the DOS APIs and get off of `PROGRA~1`, you need to handle spaces properly. Having a bunch of the standard paths (see also `My Documents`) include spaces meant that software vendors shouldn't have been able to get away with using Windows filename APIs without supporting filenames with spaces - everything would obviously break almost instantly.

Doing that, Microsoft made sure that there was no reasonable way for an ISV to claim that they either forgot to test that use-case, or weren't going to support it, when every Windows system in existence relied so heavily on paths with spaces.

Does anyone actually _use_ My Documents for anything legitimate these days?

I just use C:\Home\(my username)\ , where apps don't auto-create and leave theri junk. I guess the C:\Home\ isn't even necessary since I'm the only one using my computer.

Same thing with Linux though, my home directory is cluttered with dotfiles and hardcoded paths because people either haven't bothered to look for a standard (XDG exists) or give you the middle finger and choose to do whatever they want anyway. Snap is probably the worst of these products.

You're pulling in a 400 megabytes of dependencies the second you touch cross platform GUIs, a library to automatically create and pick the right appdata folder really isn't that inconvenient at this point.

If at least all new config files etc would be put in ~/.config/ et al, but there is still new dotfiles being dropped in ~.

Looking at you, wget and your .hsts-something file.

I went through https://wiki.archlinux.org/title/XDG_Base_Directory#Partial and set all the bullshit env variables in my .profile to force some programs to comply (not that it fixes everything).

I've seriously considered writing a system LD_PRELOAD that catches resolved paths and redirects them to a directory of choice.

I like this! It the type of chaotic good we appreciate on HN ;)

But I think it would be easier if you instead sent MR to offending projects

Sadly, most of the software I use already has a link in the list to an issue someone made about XDG support and often developers don't care, actively choose not to respect the user's preferences (snap, for one) or don't want to change the location out of fear of breaking compatibility. Those MRs would probably become forks in practice.

Good news! This already exists as libetc[0]

[0]: http://ordiluc.net/fs/libetc/

Amazing, I love it! Thanks for the link!

There is a great solution! Give Linux a try. /s

In all reality, very OS has their quirks, and I see this _all_the_time_in_Windows_!

My $HOME in WSL is also a wasteland.

>Just put your data in $env:APPDATA, damnit!

Sincerely: Hell no. Nearly everything (mostly Electron) nowadays wants to dump everything into %appdata% and it is infuriating.

I can never have enough space on C: anymore. I finally relented and bought and allocated an entire 2TB drive to just to C: on my newest machine because of "%appdata% All The Things" nonsense. C: otherwise doesn't need anymore than 100GB to house Windows and a few other system-related odds and ends.

%appdata% isn't even where All The Things(tm) should be going by default anyway, %appdata% is for storing user-specific program configuration data only. The Windows-provided default directory for program installations is C:\Program Files, and files resulting from program use should go in the program's own install directory or into an appropriate directory under the user directory (eg: My Documents) by default.

Windows allows full access to the file system and all the benefits that entails for a reason. Just ask me where I want the damn thing installed/saved rather than just shitting all over %appdata%.

Why does this frustrate you? Do you look at your home directory as a carefully-tended garden, and the translucent dotfiles are cluttering it up? Why are you attached to your home directory like that?

Also, this is why the registry was invented, but as we all know, that's a "bad idea" (because it's an old idea, which makes it bad, because being old makes ideas badder).

A few months ago, I got a kernel hang when accessing files on a shared "image storage" drive mounted using ntfs3 (forgot what I was doing, maybe using DigiKam), which surprisingly wouldn't respond to alt-sysrq. I ended up switching back to NTFS-3G.

I also tried using ntfs2btrfs with WinBtrfs, but ntfs2btrfs not only had a segfault in the application, but when I fixed it, it generated a corrupted btrfs filesystem that would enter read-only mode within a few minutes of having DigiKam accessing images on the btrfs storage drive.

Why on earth is that a driver-level feature?

Where else would it be appropriate?

Either you give every application special knowledge of both dotfiles and NTFS, or you give the filesystem special knowledge of dotfiles.

Hypothetically you could write a special daemon that listened for all filesystem events on NTFS filesystems and set this attribute for any created dotfile, but that's incredibly roundabout for one feature.

> Where else would it be appropriate?

In the file picker and "Files" utility. "Files" already has a "Show hidden files" setting. Most applications now use a common file picker, because the file picker now has some security privileges - it can give the application the privilege to access a file otherwise not allowed.

Unfortunately, the file picker doesn't create most dotfiles, and the point of the "hidden" attribute is to make them invisible on Windows which doesn't have this logic baked in.

I understand the logic of having display functionality in the file picker, but what are the chances that we can collectively convince Windows Shell32 developers that explorer.exe needs to display files prefixed with dots as hidden?

I think we're both on the same page that it's a hack, but it's a practical hack in that it avoids the need to involve unrelated third parties.

I think the former is a reasonable solution. For example, Nautilus will honour both filenames starting with a ., and all files listed inside a file named `.hidden` in a directory. Adding a third mechanism to hide files is not a big stretch.

It's not a reasonable solution because you, nor I, nor the collected users of Linux, have access to or authority to change every Windows program.

Microsoft engineering have different marching orders than hands-on 0-transparent-0 OS coders, and this yet another indication of it IMHO

Where are Microsoft engineering involved in this? It's developed by Paragon Software for Linux.

On Linux, the general expectation is that files starting with a dot are invisible. This started out as a bug in an attempt to hide . and .. from the output of ls but became industry standard for Unix derived systems.

This is noot the case on NTFS (and probably other filesystems) as it has its own way to mark a file as hidden. This is a workaround to bring Linux behaviour into a non-Linux filesystem by adding the hidden flag based on the file name rather than the file attributes.

I'm sure this will fix all kinds of compatibility issues with applications using lower level APIs for file access and enumeration. It's a non-standard workaround, though, so it makes sense to put it in a mount flag.

Linux pays no attention to the hidden attribute. But when you create a dotfile under Linux the clear intent is that the file should be hidden. Linux tooling knows to hide dotfiles by default, but Windows doesn't. So, this patch sets the hidden attribute when you create a dotfile under Linux. If you then mount the same filesystem under Windows the dotfiles will be hidden. Given that Linux userland tooling knows (in general) nothing about NTFS, and this hidden attribute is NTFS-specific, where would you put it other than in the driver?

I would not promulgate this ancient Unix hack. It was excluded on plan 9 for this exact reason as the Unix creators knew it was a mistake and corrected it.

Furthermore, the idea of hidden files seems silly to me as they are mostly used to hide dot files that should be in a directory and not carelessly tossed into the root of a users home directory.

It saddens me we pollute newer software with ancient hacks simply because no one wants to fix the root of the problem.

does the existence of a MSFT_PRIVATE partition type in the Ubuntu 22.04 installer also sound "silly" ?

That sounds quite pleasant, but I wonder who is storing their home directories on NTFS?

I considered this in the past. Linux support for NTFS is much better than any of the mechanisms to read ext3/4/btrfs/etc on Windows/Mac. Your options for a Windows/Mac/Linux portable filesystem are basically exFAT or NTFS.

That said, now I use LUKS on Linux and Bitlocker on Windows, I've mostly just given up on a shared filesystem

How do the Linux-specific properties of files get stored on NTFS (e.g., file owner/group/mode?)

NTFS ACLs are much more powerful than Unix permissions. Just map users to users, groups to groups, and you're done.

How? ACL subjects are identified by SIDs not by UID/GID.

As I said, set up a mapping between users/groups and the corresponding notion in NTFS. An UID/GID will correspond to an SID and vice versa.

ntfs-3g (is that still how Linux machines are mounting NTFS read/write? it's been a while since I've done it) by default just doesn't do anything with the NTFS permissions and makes everything have a shared owner/group/mode as seen by Linux.

You can opt in to having it store Linux permissions in NTFS's ACLs, and there's an option to have a file that maps between Linux and Windows user IDs to allow for interoperability.

> is that still how Linux machines are mounting NTFS read/write

ntfs-3g is a FUSE based driver (and not much maintained anymore), Linux now has a fully featured kernel driver with ntfs3 - which is also what TFA is about.

https://www.kernel.org/doc/html/latest/filesystems/ntfs3.htm...

Oh yes, of course, so it is (what the article is about). Oops.

Documentation for ntfs3 is a little thin on the ground.

I have not tried personally (have no Linux installed), but from docs it should be quite possible to mount your partition of Linux with WSL2. Google for "WSL mount".

May work on latest Windows 11 only though.

Does that mount the FS into the WSL2 VM or into the Windows host? I'm guessing it's the former

Yeah, but as WSL is exposed back to Windows, this should make it accessible too. Not tried myself though.

I guess nobody ATM, but if the support for NTFS becomes really good then why not use it as your root fs? It's truly battle hardened (the fs, not this particular implementation necessarily), and it is really nice to use an fs that is supported in both Linux and windows if you're dual booting.

Isn't NTFS incapable of caching as well or something due to extra layers of indirection, and that is why git is so slow (10-100X) on windows? Or is it another part of windows that slows it down?

(Edit: seems to be a combination https://github.com/Microsoft/WSL/issues/873#issuecomment-425... )

"NTFS really isn't that bad" -- https://web.archive.org/web/https://lca2020.linux.org.au/sch...

NTFS has poor performance with lots of small files.

That's Windows VFS layer. If you bypass it and read disk structures directly it's wicked fast. There are many file tools that do it such as Everything and WizTree.

Real-time antivirus scanning likely doesn't help. Most Linux users are unlikely to have this running in the background.

NTFS has horrible small file performance, to the extent that if you are working on a project that has lots of small files (NodeJS for example) you can expect performance that is 2-3x worse than on other file systems.

Even unzipping archives with lots of small files takes forever, it is easy for a modern SSD to slow down to 5MB/s or less when trying to write out tons of small files to an NTFS partition.

NTFS is old and stable, but it is not fast by modern standards.

That's a limitation of Windows' VFS rather than NTFS disk format

Just to understand this, is this on read or on write?

I.e. will the driver just automatically set the hidden flag when a new dotfile is created - or will it "pretend" the flag is set when an application is querying the attributes of a dotfile?

I would prefer the first case, as that would leave you the option to manually un-hide a dotfile of you needed to.

Reply on news.ycombinator.com