Hacker News Reader

Comment by bphogan | Hacker News Reader

FTA:

> I have to provide this guidance at least 2-3 times a day so instead I am publishing it here so everyone can find / link-to this guidance.

This may be oversimplifying, but if that's the case, then there's a design problem. Either you should make it work, or you should do some better defense against it up front.

If the solution is "eh, just work out of a Windows folder", this isn't really better than using a VM. Heck, I've been using WSL since the beta, and using Vagrant + Putty gives me better results.

I was hopeful it would get better, but then I saw this post and shook my head. "Silly users, stop trying to do things we didn't plan for you to do!"

View on HN

I have to agree with you there. I've always used the 80/20 rule for prioritizing immediate fixes: If I have to take a support call from more than 20% of the people I hae deployed my software to (and in some cases, this may amount to one customer), then whatever the circumstance is, it's treated as a high priority bug.

That said, Windows Subsystem for Linux is beta and they went out of their way to make that clear -- going so so far as to put (beta) in the Add Features dialog. And this was the right move, IMO. Such a small percentage of their user base use this feature -- it's targeted at us and we're used to dealing with beta problems. They hit a large enough number of users to test the feature thoroughly, while managing expectations that it's not ready for prime time, yet.

> "Silly users, stop trying to do things we didn't plan for you to do!"

Yep, that's pretty much it. But in the context of a beta product, it's more "Here's something we are getting large numbers of reports about while we're refining the software for production release. In the meantime, don't do this."

Now, if they release this production with that limitation and nothing to prevent you from destroying your data if you modify that folder, that'd be a major oversight, but I have a feeling that'll be resolved. My hope would be that you'd be able to mess with these files using whatever tool you wish, but even adding a dialog to Explorer, and an error in PowerShell/Cmd that catches you trying to do something that'll break the subsystem would be an improvement.

> in the context of a beta product, it's more "Here's something we are getting large numbers of reports about while we're refining the software for production release. In the meantime, don't do this."

I don't see any indication in the article that this behavior is something they intend to change in order to "fix" a "problem". It looks to me like don't do this, period.

If/when this issue is resolved, the guidance will be updated. Until then, don't go futzing with files in hidden system folders.

EDIT: Misinformed, see wfunction's comment below.

I can understand their frustration, because this is probably really hard to fix.

I guess there would not be problems if Windows apps were not overwriting files by first deleting them and then re-creating them with new contents. It's easy to see why this leads to data loss (file permission metadata is data in the Unix world). You rarely see Unix apps re-creating files: there you instead you specify the behaviour in case the file exists with the flags of the open(2) call.

I guess such contract is absent / not as strict in the Windows world (disc: I have never programmed for Windows), which would make solving the conflicting APIs almost impossible: if a re-created file would inherit a previously deleted synonymous file's permissions, what happens to a program who is deleting a file exactly to get rid of its permissions?

I think MS has done pretty impressive work with the Linux subsystem. It is certainly not a trivial task, and I am looking forward to seeing more awesome stuff running on Windows.

> You rarely see Unix apps re-creating files: there you instead you specify the behaviour in case the file exists with the flags of the open(2) call.

I think your lack of knowledge about this problem is betraying you. This is outright wrong. First, look up CreateFile() in Windows; it has flags to specify things like this as well. Second, the reason programs delete files in the first place has nothing to do with the lack of such flags. It has to do with the fact that they want to write new contents but don't want to lose data in the event of an abnormal termination. If you truncate the file that you're opening instead, then you lose that data if something goes wrong. So they create a new file and replace it with the old one when it's written.

Finally, as far as I know, 'nix tools have a tendency to outright replace files MORE than Windows tools do. That's why the underlying Windows kernel API function (NtCreateFile/ZwCreateFile) has a FILE_SUPERSEDE parameter... whose entire intention is to mimic POSIX behavior:

> The CreateDisposition value FILE_SUPERSEDE requires that the caller have DELETE access to a existing file object. If so, a successful call to ZwCreateFile with FILE_SUPERSEDE on an existing file effectively deletes that file, and then recreates it. [...] Note that this type of disposition is consistent with the POSIX style of overwriting files.

https://msdn.microsoft.com/en-us/library/windows/hardware/ff...

Exactly. Most file systems have an atomic move, that you can leverage into an "atomic write".

Thanks for correcting.

That makes it sound almost like this could be actually fixed by swimming deep enough to the NTFS driver in Windows.

There is such a thing as NTFS tunneling, if that's what you mean. Not sure what you have in mind if you mean otherwise.

NTFS tunneling is a hilarious feature.

To those who are not well versed in this:

This was a hack for 2 things that exist in the universe:

1. Apps that save their documents by "delete document, re-create document by the same name".

2. The fact that said apps (especially when you consider that they might be written for Win3.1 or DOS) could be accessing the documents by 8.3 names (eg. MICROS~1.TXT)

What ntfs.sys will do is: for some period of time after deleting a file, remember the "long filename" of a corresponding short name, and re-hydrate it when you re-create. So when you delete MICROS~1.TXT and create a new MICROS~1.TXT, ntfs.sys will remember that this file is also called "Microsoft.txt" and re-create that name.

I would go so far to say (though I've met few people who grok this, as Windows filesystems is pretty rare as a specialty these days) that delete-and-recreate is a huge antipattern and recipe for failure on Windows.

The reason: Delete behavior.

In NT, when you delete a file and handles are open, the name sticks around until all handles are closed. Any attempt to re-open (including re-create) with the same name will fail with STATUS_DELETE_PENDING while this is the case.

Combine that with the fact that most operations in NT (including delete or rename) happen via opening handles to the file first and you have lots of chaos and race conditions.

So the delete + recreate pattern is very likely to bite you later as a "re-create fails randomly".

What's your alternate recipe for success?

I don't think I have one unfortunately. It depends greatly on the scenario.

Usually I'd say come up with a unique name if you can. Otherwise I'd say try to overwrite (possibly overwrite via rename, rather than CreateFile, to avoid the data loss you mention).

I've even written some code to rename to a unique name first, then delete.

OK, so let me explain why your reasoning is wrong.

The problems here only occur if your app is modifying files that do not "belong" to it. For apps that store config files, databases, etc., these problems should not arise, because those files should only be touched by those apps, and (a) the app is already aware what metadata the file should have and is the one maintaining control over the files, and (b) the app can make sure that it is never using a file that it is trying to delete. If the user is messing with the app's files, then the user is just as responsible for not preventing their deletion as he/she is responsible for not corrupting their contents. You can't blame the deletion behavior for that; the responsibility for proper care rests entirely on the user, and having a different delete behavior doesn't fix the core problem.

For files that don't belong to the app, the situation is entirely different. First, apps should not be deleting files that don't belong to them without user consent. And if the user is consenting, the user is responsible for ensuring this does not cause problems. Second, those that modify files they do not own are responsible for all aspects of this, including preserving metadata. This is of course difficult and quite a burden, but at this point, it is considered a bug in the application if it does not do this correctly. Again, the app is the one mucking with files that it does not own, so the app and the user together are responsible for maintaining consistency, not the OS.

You're funny. Telling me my years of experience of this behavior biting me in the rear end is "wrong".

There isn't even such thing as "my files" and not. Are you aware that many people run antivirus products that sit between you and the filesystem and might choose to also open "my" files in an asynchronous fashion based on my own actions? That such an antivirus product has been built into Windows for some time now? (msmpeng.exe) Suddenly that file you thought it safe to have exclusive access to... it wasn't.

Even controlling your own actions is hard and yields unexpected results all the time. Almost any time setting dwShareMode to something other than 0x7 and I saw it bite me in some unexpected way due to things my own process is doing, that I did not think of ahead of time.

Once you have Windows filesystem code doing this at any appreciable scale you will see this. It takes enough of seeing sharing violations and delete pending errors time after time to realize these are just dangerous patterns on the platform, avoid them and move on.

I didn't say your experience was "wrong" though? That doesn't even make sense; your experience is what it is. I only said your reasoning about your experience was wrong, i.e. you should be blaming the applications and not the OS (whether they're made by MS or anyone else).

Regarding antiviruses: yes, of course I am. That's why they should be opening the files with FILE_SHARE_DELETE, and properly handling the edge cases. Unless you're telling me it's impossible (which I strongly suspect isn't the case, but then again I've never tried to write an AV myself), I don't see how it's the OS's fault. This flag and the associated functionality obviously exists for a reason, right? It's not an OS design flaw if people aren't using it, is it? People can refuse to play along with anything... that doesn't mean the OS is flawed.

Sharing modes don't help with the STATUS_DELETE_PENDING issue. Nor does it let you delete a directory while children are open.

Also some AV will duplicate your handles, meaning they get the same sharing mode you asked for.

> This flag and the associated functionality obviously exists for a reason, right?

It's the kind of stuff that sounds reasonable when you hear about it, but when you see it put into practice is the source of way too many bugs.

> Sharing modes don't help with the STATUS_DELETE_PENDING issue.

Maybe I misunderstood the problem. But I think my reasoning holds regardless? To be clear, it seems you're talking about a situation where an app is trying to delete and re-create a file, and an AV is scanning the file in between, hence a recreation fails.

But with their file system filter drivers, AVs can detect such recreation attempts, so they should be able to handle it properly. So isn't it their fault for not doing so?

> Nor does it let you delete a directory while children are open.

Right, but same as above: there exists functionality to detect this, right? So if it isn't used, whose fault is it?

Actually it's fairly common when making changes within a file, especially a text file, to read from one file, write the changed data stream to a new file, then do an atomic rename of the target file to the name of the source file.

> You rarely see Unix apps re-creating files

Vim's default behaviour is to save via complete re-creation and atomic rename. I wouldn't be surprised if other classic editors used the same technique.

That is also what "sed -i" does.

> because this is probably really hard to fix.

The alternative I can think of is spreading hidden metadata files all over the place, à la Apple with their .DS_Store and AppleDouble files that everyone always complains about. That comes with its own set of problems (moving files around will lose the metadata when they're no longer in the source directory along with the metadata)

It's easy to see why this leads to data loss (file permission metadata is data in the Unix world).

Completely disagree. Considering that some file systems which have been around for decades don't even have this permissions metadata, I would say that the file contents are the most important to users. MS should absolutely give a warning that the extended metadata can be lost, but the "If WSL can’t find Linux file metadata for a given file, it assumes the file is corrupted/damaged" behaviour noted in the article is completely contrary to user expectations.

The filesystem model in Linux and Windows is not identical, but there is considerable overlap. It therefore makes perfect sense --- and is expected --- for the overlapping parts to work.

> Remember: There’s a reason we gave the %localappdata%\lxss\ folder ‘hidden’ & ‘system’ attributes

They already did that, but as usual "with great power comes great responsibility", so if you ignore the safeguards installed .. well .. bad things can happen.

Indeed, even with my level of tinkering, I almost am never dealing with system attribute files. The cool thing is that Microsoft ALLOWS you to do anything with your computer, they don't hold the sort of superiority complex other companies do about telling you what you can and can't do on your own PC[0].

So yes, if you decide to grant yourself permission to see and modify those files, you can give yourself a bad time. Or maybe you could develop an application that handles them intelligently, after researching how they work extensively.

[0]Someone's gonna ask: With the right amount of registry and file tampering, yes, you can absolutely render automatic updates and telemetry nonfunctional. Results may vary, but it's definitely doable.

I have to assume you are alluding to Apple when you say "other companies". If that's correct, it's relatively trivial to disable the protections on system files in all versions of OS X / macOS. Furthermore, automatic updates and Apple's less invasive version of telemetry are both opt-in, not opt-out and certainly not requiring the magic mojo Windows requires to do so. Sure, iOS is a police state in comparison, but you explicitly stated "on your own PC", not mobile device.

I was actually thinking of Google at the time. Chrome OS devices, for example, wipe themselves if a system file is changed. The new Pixel needed an exploit to root it. Companies like Google (and yeah, Apple too to some degree) believe that you should only experience your own computer the way they intended it to be experienced.

For Chrome OS, all you need to do is enable developer mode to fix that. It's a supported feature in the OS and documented in detail on Google's website.

I think you are being a bit too harsh here, there is a way in which it does work, all of your files are on the "windows" side. For those of us who have lived in both camps (windows and Linux) the difference in file meta-data is extreme, it isn't an "easy" problem at all. Windows has metadata bits for all sorts of things that Linux doesn't which gives it much greater flexibility but means that the equivalent of your umask on Windows doesn't translate. It does however do everything Linux / POSIX needs so the inverse is doable.

"for those of us who have lived in both camps"

I live in both camps. This is the same attitude I took issue with in the original article. It's condescending. "I have to keep telling you to not do something because we didn't plan for it."

I'm not saying it's an easy problem. But I am saying that blaming users for something doesn't give me confidence.

That's like having users start deleting things out of your /usr/lib and then complaining the fault is clearly with the OS designers for not securing that folder sufficiently.

That's a valid position though. Maybe iOS is for those folks.

the system attribute is that better defense, changing explorer's options to display such files results in the following warning

https://i.imgur.com/uUaJEZI.png

if it was done via a VM, people would just mount the VHD, edit the files, and still complain

Microsoft puts so many things behind this flag that you pretty much have to display all system files if you are a power user.

Totally disagree. I enable it on the rare occasions I need to and disable it immediately after I'm done doing what I need. This happens quite rarely (a few times a year) so, basically: no, I beg to differ.

Hmm... I'm not too sure about that. Any examples?

Show file extensions is disabled by default. I'll always go in and change that. But I can't think of a time where I've had to go into a system file before.

I've left the system files invisible at least since installing Windows 10, and it hasn't once been a problem. I can't remember if it was always this way, or if it's a recent addition, but there's a separate option for showing hidden files and folders. (With this ticked, system files remain hidden, so this one is a much easier tick to recommend, if you're technically-minded. You get to see %USERPROFILE%\AppData, which you'll probably want, but not stuff like hiberfil.sys and boot.ini, which you need much more rarely.)

My experience is that this was arguably true in the NT/9x days, wasn't really true in the XP days, and is quite false with Windows 7 onwards.

What exactly has caused you to need to unhide those files this decade?

Isn't Windows' hosts file located in a system folder? Not that I change it so often anymore, but when I did more web development, it was not uncommon for me to add a line there...

No, the "system32" folder is a normal folder from that perspective, i.e. it does not have the SYSTEM and HIDDEN attributes set.

Same for the "drivers" and "etc" sub-folders.

The registry hives are still system files, and so if you want to edit a particular user's registry, you need to show them.

Hidden files, sure. But system files? I can't see any reason to ever display them.

Yes, it's totally a design problem that users are fucking with files in a system hidden folder. I do that all the time, easy mistake to make.

More like editing some config file under the WSL folder with Notepad++ and discovering you destroyed it.

> WSL folder

Which has both the hidden and the system flag set. You have to change two explorer settings to even access the directory, one of said changed gives you a prompt warning you that it might be dangerous to enable it.

This is the bullshit I really don't miss from the Windows world.

*nix: "Go for it, edit what you want, just be aware you can fuck up"

Windows: "what the fuck are you doing in the special area? get out!"

The only difference in assumption is that Linux users can safely assume most Linux users kinda know what they're doing. This is not true for Windows, where the default assumption is, you probably don't know what you're doing. ;) When you do know what you're doing, nobody complains about you going in the crevices.

One of my favorite Windows v. Linux things is when Linux users try to work in Windows, and try and do things that would be perfectly normal in Linux, and then wonder why their Windows box is so unstable. I once ran across someone who tried putting their Windows\system32 folder into a version control app. When their Windows registry exploded spectacularly, their response was that this was a normal thing to do on a Linux box.

Windows and Linux take very different engineering approaches to things, so skills developed in one don't necessarily translate well to the other.

That sounds obscene even to my Linux ears... why would it be a good idea to version control the system directory in any OS?

Presumably to be able to track changes and undo them if a set of changes causes a problem? I was kinda skeptical. But I certainly wasn't blaming Microsoft, like he was, for his system being unstable. =P

The person may have thought that \Windows\System32 was the moral equivalent of /etc , and expected to be able to do with it what people do to /etc with tools like etckeeper.

And, of course, ZFS boot environments are the same sort of idea to an extent.

Do you remember the hue and cry when deleting things from efivarfs could brick your machine, and it was somehow systemd's fault for making these files available and not locking out end users for their own protection? https://github.com/systemd/systemd/issues/2402

To be fair, that broke the paradigm of your OS not being able to brick your hardware. This had rarely been an issue before, and most people would have assumed it wasn't even possible.

Sure. But I feel like I remember warnings in the mid-'90s about bad modelines in XF86Config being able to cause smoke to come out of your monitor....

Isn't that the whole point of this post though? They've allowed you to edit files marked as "system files" despite a multitude of warnings, and now so many people have fucked up doing so that they've issued a stronger warning.

What's the solution? Put less warnings? Then you'd get even more people destroying their data. Put more warnings? Then you'd get more people complaining that "Windows doesn't let you edit what you want!".

It doesn't really seem like they can win.

I'm more talking about the culture between the two types of OS than this specific occurrence. The whole attitude of "well, why the fuck were you in there anyway?" is not something I've encountered in *nix.

Windows at least has the decency to warn you that you're about ready to trash something. Every -nix ever will let me blow /sbin/init away with a carelessly placed file operation without so much as a "hey".

Uneducated users should be responded to with "WTF were you doing?" as computer is a precision machine and it is not unreasonable to expect some basic level of literacy.

    vacri@thingy:~$ rm /sbin/init
    rm: cannot remove ‘/sbin/init’: Permission denied

I don't seem to be able to blow away /sbin/init.

If I've elevated myself to be a global administrative user, then I should be able to do global administrative things, without being treated as an 'uneducated user'.

> it is not unreasonable to expect some basic level of literacy

I think it's perfectly reasonable to expect some level of literacy for administrative-level tasks, and that's what nix does.

Isn't the entire point of the post to say that if you edit these files, be aware that you can fuck up (and here's how you can fuck up)?

I'm responding to mona3000, not the post.

they aren't telling you to don't touch them, they're telling you to don't touch them FROM THE WINDOWS SIDE. Start a bash instance and use vi/emacs/nano/ed/whatever to edit it from the Linux side.

You might want to understand my point before you start 'shouting' at me.

I'm sorry, I didn't want to appear rude, I just couldn't remember how to italicize text here on HN and tried to emphasize those words. By the way, here they're just telling you to stop messing with those files in a certain way because it isn't a supported scenario in a beta software, also giving you the reason why it isn't supported. By the way, I bet that thing isn't among the top 10 priorities.

Maybe they're working on that too. Writing a blog post to warn people about a problem doesn't preclude fixing it.

Short of literally removing admin access to the box, how would they go about fixing this?

One way would be to run a user level nfsd on the Linux side and mount it on the windows side as an nfs share. Or if they are morally opposed to that running a user level samba server and mounting it as a CIFS share. Both of these solutions allow you to access Linux files from windows apps on their native storage layer without corrupting them or the system they are running on.

You might be able to accomplish that with a system service that runs in the context of the LXX shim and a local (in memory) network connection.

"I've been using WSL since the beta, and using Vagrant + Putty gives me better results."

If WSL can't do the equivalent of Vagrant-style vm.synced_folder, how is anyone expected to use it for a real-world workflow?

I've found the combination of Vagrant/Virtualbox with the Git command-line tools (using MSYS2) to be an excellent cross-platform working environment. Really, it's been my daily workhorse for the past two years.

Also, there is a not-free program called SecureCRT which is much more usable than Putty.

I like mobaXTerm it is pretty useable and have a free version.

From the article:

"Remember: There’s a reason we gave the %localappdata%\lxss\ folder ‘hidden’ & ‘system’ attributes "

So if you go messing about in hidden system folders, you get what you deserve.

One could also argue that an editor that loses extended file attributes when editing a file is broken.

I would have preferred msft to have posted an article explaining that files have attributes and editors must not loose those attributes when making edits.

Files on Windows don't have attributes. Files on an NTFS volume may have extended attributes associated with them but you'll need to go out of your way to find them and doing so has never been a requirement of working with files on Windows. Not supporting an obscure non-mandatory thing hardly constitutes "broken".

An editor which just kept the existing attributes would still end up with incorrect time stamps - so now all Windows apps need to know how to handle Linux file attributes? Nope - because these files were never meant to be edited by Windows apps. And it works best that way.

I didn't realize extended attributes were such an obscure feature. I haven't been on windows for years, yet knew about this feature.

I'm on OSX and every now and then see how apps are making interesting use of file attributes (e.g. tags, or remembering the cursor position of a text file when re-opening). So, I'm honestly surprised Windows isn't doing anything with this useful OS feature.

Reply on news.ycombinator.com