Hacker News Reader

Comment by mehrdadn | Hacker News Reader

I mean, you could say anything is subject to change. The "documented" <-> "stable" mapping isn't a bijection. CreateFileTransacted was documented for kernel32 but that didn't stop it from being claimed as deprecated. And CallNtPowerInformation was documented and yet some of it simply doesn't work anymore.

Also, Microsoft didn't necessarily document everything, but they did provide .lib files (and often headers) even for "undocumented" APIs in these libraries. Breaking these APIs would break programs they previously provided SDKs for.

The reality is, so many third-party applications depend directly on ntdll APIs (even Chrome) and some of them literally cannot work with kernel32 stuff (boot-time partition managers, for one), so as far as I'm concerned, it's about as much set in stone as any library could be.

View on HN

Yeah. Direct dependence on ntdll occurs in the wild and it would break stuff to change it. It can also break MS's own code and potentially create massive engineering hurdles for them that are completely unnecessary. But they have always warned people in documentation that they don't consider it a stable interface to the same extent as Win32.

In my experience a lot of the NT APIs are nicer, better thought out, more direct.

Using the NTDLL calls breaks stuff because NTDLL doesn't understand anything about the overlaid subsystems. NT was originally designed to support multiple subsystems in addition to Win32, like the Interix (POSIX compliant) subsystem and the OS/2 subsystem I believe. I think (I could be wrong on this) that this was possible by switching the system call table depending upon which process was currently active (doable by switching the system call vector).

IIRC the syscall table doesn't get swapped out. Each subsystem translates its calls to NT API calls. For example, user32.dll and kernel32.dll are a part of the win32 subsystem and eventually end up calling NT APIs in ntdll.dll. It's possible for a process to have no subsystem, these are called native NT process and the only dll loaded by default into their address space is ntdll, csrss.exe is an example of this.

The layer underneath that they build subsystems on is always the system call interface (ntdll), not the syscall numbers themselves.

It's a political thing. Microsoft isn't going to break themselves. It's more a question of, if you go against all the warnings and use NTDLL in the non-recommended way (i.e. having it be a hard dependency w/o fallback) Microsoft can and will fuck you if they don't like what you're doing. Just as recently as a few months ago, I saw a startup in Sunnyvale called Crossmeta basically throw in the towel, because they built their whole product around this one NTDLL fork() hack, which had been around for nearly decades. Then MS rolled out a security update, and poof, it's gone.

Wow, cool story. The part I find surprising is -- there was a fork() hack on NTDLL that actually worked until recently? I know tons of people had tried to get it to work (including the Cygwin folks and also including me) and faced failure after failure!

Technically the Windows native layer supports fork()-like abilities, but the problem is that the Win32 subsystem and dependent layers (GUI, etc.) don't have fork() copy-on-write capability. So even if you were able to fork a process, you'd still have to deal with duplicating all of the stuff built on top. In fact, the old POSIX compatible subsystem layer utilized this functionality to actually provide a compliant fork().

Of course this all ignores the fact that Win32 processes are heavier duty than Linux processes (though I don't know if that's due to Win32 subsystem overhead or not). Look at any benchmark and you see an order of magnitude difference in process creation times. You're much better off creating threads instead.

WSL1 also uses that functionality

Huh? WSL1 doesn't use NTDLL, it implements its own stuff.

I wasn't specifically talking about any ntdll call, but in one of their "deep dive" videos on Channel 9 they state WSL1 supports fork() because the NT kernel natively supports it, it just isn't exposed on the normal API surface

I think you got confused. The "native layer" referred to NTDLL in the comment you replied to. I think(?) that's what the earlier POSIX subsystem was built on. That's why I said WSL doesn't use that.

Pretty much all CPUs with MMUs since MULTICS inherently support fork(). So for NT it's a question of prohibiting the behavior. Microsoft's research department even writes papers about how they disagree with fork(). I was truly impressed by the work they did implementing Linux ABI in the NT executive for WSL 1.0. Big change in policy. It's going to be sad to watch it go, but might be for the best. A really well-written ghetto for Linux code is still a ghetto.

This is a weird statement. The MMU doesn't natively do a fork(). The kernel needs to implement the copy-on-write-fault. I.e. all pages marked read only and the write fault handler needs to realize a given address is COW and perform a lazy copy.

Let's consider looking at this in terms of data structures, and ignoring the APIs. Your CPU has this:

    register char (*(*(*(*ram)[512])[512])[512])[4096] asm(cr3)

Your CPU resolves every pointer memory access through those tables under the hood. It's an extremely powerful data structure. It can let you allocate linear memory like a central banker prints money. But if you're a Windows user, then only Microsoft is authorized to access it, and they don't want you having the central banker privileges that are needed in order to implement fork(). That's the way the cookie crumbles.

It seems you replied to the wrong comment?

Looked up Crossmeta just now and it appears that there “threw in the towel” just on their fork() emulation, not on the whole project that does quite a bit more.

http://crossmeta.org

Reply on news.ycombinator.com