The Definitive Guide on Win32 to NT Path Conversion (2016) | Hacker News Reader

The Definitive Guide on Win32 to NT Path Conversion (2016)

googleprojectzero.blogspot.com

113 pointscamtarn39 comments

Windows paths can be really infuriating. We constantly run into problems with the 255 file name length limit and even if you use the APIs to create longer paths then Explorer or other apps can't deal with these names.

It's hard to believe that after all these years this is still a problem but I guess a clunky foundation doesn't prevent one from being successful.

The ugliness of the windows API never ceases to amaze me. Functions with names like RtlDetermineDosPathNameType_U and return types like PCWSTR just make my eyes bleed.

My personal favorite is PCZZWSTR: a const pointer to a contiguous WCHAR array, where each substring is NUL-terminated, and the array itself is also terminated by a NUL. Thus (I assume) the ZZ--zero zero. I've only ever seen it used in the SHFILEOPSTRUCTW struct.

That reads like one of the instruction mnemonics from https://twitter.com/x86instructions

>SHFILEOPSTRUCTW

now that takes the cake

Nah. That would be INITCOMMONCONTROLSEX. And the related LVM_SORTITEMSEX, DDSCAPSEX, and of course MPR_CREDENTIALSEX_0.

With Microsoft's use of ALLCAPS and using an "Ex" prefix, this was all but guaranteed…

How about SHITEMID in the shlwapi DLL? That's supposed to be SH-ITEM-ID.

if someone ever figures out the meaning of that phrase it's probably going to summon some cosmic horror that will be the end of us all

I've had some interesting experiences with SHFileOperationA on Wine. Star Citizen's updater wasn't working properly at one point due to it.

Multi-strings are also pretty common in the registry, see REG_MULTI_SZ.

I feel like that construct, sometimes called "double-null terminated", is often inaccurately described.

I prefer to think of it as a contiguous block of null terminated strings terminated by an empty string. (Just as '\0' terminates a C string, the empty string terminates such a list.)

That's not an API. The article talks about some ntdll function, not supported by MS, and not documented by MS.

The ugliness might be deliberate, to make developers conclude "that must be some Windows internal BS, probably undocumented" when they see it in debugger, profiler or other places.

> The ugliness might be deliberate, to make developers conclude "that must be some Windows internal BS, probably undocumented" when they see it in debugger, profiler or other places.

Not at all. It is consistently named with internal NT APIs that are used like that intentionally. The people working at that layer follow such naming conventions without irony.

> consistently named with internal NT APIs

I know, but these are mostly not part of WinAPI either. The public API surface has very good names, WriteFile or CreateDirectory is IMO better than POSIX write or mkdir.

My favorite Unix function has to be "creat". It doesn't even make sense given the limitations of the systems at the time.

It definitely makes you wonder whether the 'E' key was broken on the machines at Bell Labs.

In all seriousness, Ken Thompson was asked what was the one thing we wanted to change in Unix if he could, and his answer was that he would spell "creat" with an "e"[1]. And funnily enough, he actually did in Go[2].

[1]: https://unix.stackexchange.com/questions/10893/what-did-ken-... [2]: https://github.com/golang/go/commit/c90d392ce3d3203e0c32b3f9...

I don't know if they're good. WriteFile and CreateDirectory are fine, but when I think of the Windows API, I think of much longer function names. I had to use ConvertStringSecurityDescriptorToSecurityDescriptorA() once (and yeah, I had to use the ANSI version explicitly.) Another C/C++ developer might have called that function sec_sddl_to_desc(), which is much better IMO.

I know I'm probably in the minority, but I vastly prefer the long-form name in your example. Modern IDE's (especially Visual Studio with its Intellisense) can be incredible with their autocomplete, superseding the need for short-form obtuse acronym-laden function names that you need to remember.

So let's assume there's no time difference between writing the long or short form of that function. There's only one other direction to consider, which is reading.

Try to approach the following question without personal bias. If you have never worked with either API, which function name would be more self-explanatory for you? ConvertStringSecurityDescriptorToSecurityDescriptorA or sec_sddl_to_desc?

One problem is that there is no one Windows style. The style guidelines, if you can call them that, depend on a few things like age and provenance of the component (some examples of pieces with distinct styles: filesystem, registry, GDI, NT API, COM). And sometimes you see a mix of these, or a component come along with its own variation. (SDDL and other security APIs is a good example of a unique style.)

But sometimes people confuse unfamiliarity with ugliness, being bad or wrong, and having worked with a lot of these stuff I feel this is often a mischaracterization. Unfamiliarity doesn't always make it bad.

Shoutout for calling himself a "path'ological reverse engineer"

> we can only store at most 215-1 characters, which just happens to be 32,767

I think the formatting ate the superscript here.

Yeah, I think it did.

> Somewhere in Explorer something is passing one of these shell names to a file API, which is interpreting it as a Drive Relative path for the ‘:’ drive.

Isn't this just a pure bug? I'm quite surprised such an intimidating behavior still exists in today's Windows...

Gotta love how there's a whole chunk of complicated compatibility code to emulate the old DOS-ism of having per-drive current directories.

> Fortunately this doesn’t seem to work in the Explorer shell, but it does work when passed to native APIs such as CreateFile.

CreateFile is Win32 API, not Native API. NtCreateFile would be Native API.

And yet with all this, I still prefer Win16/Win32 paths and system layout as they are implemented for user needs compared to any UNIX/BSD/Linux pathing and system structure.

Windows paths is full of WTFs:

- The main driver is named C:\ because DOS used A:\ and B:\ for floppy disks! C'mon, floppy disks were not used as the main disk since god knows why (I think even MS-DOS itself was already supposed to run from HDDs and not floppy disks).

- I can't have a file named LPT1, COM, AUX, PRN, NUL, etc. This would be fine, however I can't even have a file named aux.ext, for example. I know this is for compatibility purposes with DOS, however the fact that the error message is terrible does not help.

-255 char limit. I know that you can use an API that allows more characters than this, however this does not work everywhere (last time I remember having problems to access those files in Windows Explorer; I think they fixed this in the latest versions of Windows 10, though).

- Some common characters are prohibit in filenames, like ? or :. Unix has some limitations too, however the Windows are much more ridiculous.

- In 64 bits systems, C:\Windows\SysWoW64\system32 is for 32 bits DLLs, while C:\Windows\system32 is for 64 bits DLLs. I really is curious why Microsoft choose this one, it is not like they needed for legacy software [1] (since 64 bits were new systems not old ones).

I mean, most of the Windows WTFs are for compatibility purposes, however Unix has a much more sensible path rules even if you consider that Unix has much more history than Windows. There is much less WTFs in Unix paths (like /usr/bin vs /bin, or /opt vs. /usr/local).

I can't really understand what is going on with Windows paths. Unix paths are far more sane to me IMO.

[1]: If someone can link to me the reason why I would be thankful. I know that Microsoft engineers are not stupid, I just find it really curious why C:\Windows\system64 was not used, for example.

I would guess that the system32 issue is they wanted you to be able to merely switch your compiler flags to 64-bit, without having to audit your entire codebase and build system for hard-coded references to system32. But whatever the reason, I'm sure they could have implemented something of equal or lesser complexity that would have also made sense.

Just run 'grep -r'...

Oh wait, Windows doesn't have grep.

[1] was also all about compatibility. It's quite amazing the lengths Microsoft went through to preserve compatibility during the 32 to 64 bit transition, and how well it continues to work to this day. That all came at the expense of confusing the hell out of mere users or developers looking back years later.

It was important that programs compiled for a 32-bit windows could run without modifications on a 64-bit windows system. While some amount of 32-bit application recompiles or rewrites was inevitable, a surprising number of them could get by without modifications thanks to Microsoft's emulation & compatibility efforts.

"wow64", or windows on window64, is an emulator of sorts to run 32 bit programs in a 64-bit windows environment with various behavior changes designed to trick 32-bit applications into thinking they were really running on a 32-bit system. The syswow64\system32 thing is one such effort.

If I have a 32-bit program running I may have hard coded paths to files or libraries my program needs. If my program does a fopen("C:\Windows\System32\whatever", mode) expecting to find a dll and it's not there or cannot be opened because that's now where 64-bit dlls live, my program will blow up.

So, when running in wow64 mode, the path "C:\Windows\System32" resolves to "C:\Windows\SysWow64\System32" when running a 32-bit program on 64-bit Windows. When running a 64-bit program, it resolves to what you'd "expect", "C\Windows\System32". That's a strange name for a 64-bit system folder, as you note, but imagine if it were something like system64 -- then if I want to support both 32 & 64 bit systems, I'd need different paths and logic in my program (and it's installer, and in all my tests, and ...) to switch between the two areas for 32-bit and 64-bit systems. By having the folder use the same name (from the program's perspective) everything gets a lot simpler.

This same approach happens when you are looking at Program Files vs Program Files (x86), and in the registry, where HKLM\SOFTWARE\ is 64-bit, while HKLM\SOFTWARE\WOW6432Node\ is for 32-on-64.

This emulation is so complete that it is actually difficult for a 32-bit program with awareness of Microsoft's emulation to actually find the files in SysWow64\System32. If for example, I had an installer program that wants to install my software for both kinds of system, it is difficult if you don't follow best practices and use 2 MSI installers for each bitness and wrap them in a bootstrapping installer program.

I learned all this the hard way, by shipping a 32 & 64 bit windows library. We offer both 32 & 64 bit versions because it has to work with old 32-bit software, even to this day. It is quite confusing but if you squint hard, understand their reasoning and follow it through all the use cases it makes a decent amount of sense.

Of course, a lot of those problems come up because your installer is dropping stuff in system32. Stop doing that, please.

My problems were more with program files and the registry; I wouldn't dare go near system32. But the effects are all the same and they make sense holistically for Windows, given their long standing effort to preserve the compatibility of software written on their platform.

That said the same library worked on a variety of Linux and Unix-like systems too, and packaging & installation process there was much more straightforward and predictable for me, the developer.

You can't download and extract Unix make because of one of these reserved filenames (it's either CON or AUX, can't recall).

The short answer:

"The 32 is just part of the name and doesn’t mean anything." -- Raymond Chen

The long answer:

"There are quite a number of existing 32-bit programs that hard-code the System32 path rather than calling the GetSystemDirectory function. When these programs are recompiled for 64-bit Windows, they will still try to access the System32 directory, expecting to find 64-bit files (because the program is now compiled 64-bit). Paths written into configuration files or the registry need to be meaningful to both 32-bit and 64-bit processes, yet need to refer to the appropriate directory depending on the “bitness” of the program doing the asking."

https://docs.microsoft.com/en-us/previous-versions/technet-m...

Basically System32 is the system directory regardless of the bitness of the OS. What matters is the bitness of the program being executed. If a 32bit program accesses System32 on a 64bit machine, it's silently redirected to a folder containing 32bit libraries.

I think the only kind of program that need(ed) to know the distinction and access those paths directly and deliberately would be an installer for a different-fitness program. But that was also still in the days where programs would place their files in the Windows or system directory which hopefully no longer happens.

The Unix names are more ridiculous in my opinion: /usr/: in the beginning what is now /home/ but when the space in /bin/ got too tight binaries moved simply there

FreeBSD has a much better laid-out filesystem than Linux (which is what most people think of when they complain about "Unix directories").

To your first bullet point: For the first year and a half of its existence, the only shipping model of the IBM-PC came with one or two floppy drives, and no hard disk was available at all. And MS-DOS was the operating system of choice. https://en.m.wikipedia.org/wiki/IBM_Personal_Computer

Reply on news.ycombinator.com