Python multiprocessing is different under Linux and Windows | Hacker News Reader

Python multiprocessing is different under Linux and Windows

rhodesmill.org

36 pointsgthank26 comments

I said this in a reply to another comment, but it's worth repeating. Yes, brandon is right, there is a difference, albeit an intentional one. I agree with him that having an option to force this behavior to match across platforms is desirable, however, I do not currently have the time to implement it myself.

So, in short; I agree, patch welcome or we wait until I or someone else has the time to do it.

He should stop trying to test for Windows behaviour on Linux and use a VM. Having the multiprocessing module expose an option for turning off forking would be useless, as Windows is not UNIX and there are many other subtle or not so subtle differences.

If multiprocessing exposed such an option, it would effectively be harder to write non-portable code, which seems like a good thing (to do optionally). You could then switch on "no, do a sensible fork(2)" purely as a performance optimization. Seems like a pretty obviously good thing to add.

Someone should call Microsoft and report a bug on how Windows handles child processes. This seems serious ;-)

Can anyone check if this problem also exists in the Cygwin version of Python? It's closer to the Unix source and should behave in a more unix-ish fashion. I am glad I no longer can ;-)

I always install Cygwin when I work from a Windows box. It gives me a varnish layer of civility on what I feel is a mostly barbaric environment.

That said, the POSIX subsystem NT had should be able to handle proper process forking. I am not sure how easy (or not) would it be to run Python on it, but that should also work.

NT's POSIX subsystem never supported fork(). They, um, overlooked that bit...

Actually, that's incorrect. Both the old-school POSIX subsystem and the Interix (Services For Unix) subsystem support a fork system call with copy-on-write semantics. In fact, the NT kernel itself can fork processes trivially, but this isn't exposed in the Win32 subsystem, like most of the fun stuff on NT.

I always found it weird having a bunch of Windows kernel operations that are not exposed to Windows apps.

How would you call those?

Ntdll.dll contains the userspace side of the NT API. Specifically, Zw* and Nt* contain most of the APIs that map directly into the kernel. Then there's the Rtl* stuff which provides a lot of fun functions for manipulating processes, and the Ldr* stuff for loading binaries; Ldr is a lot of fun to hook into, if you want to manipulate binaries with a high degree of transparency.

Edit: One thing to note, though, is that it's easy to subvert the Win32 subsystem by using these APIs directly. I.e. if you want to fork a Win32 process, you're going to need to not only fork it the standard NT way, but then tell CSRSS what you're actually doing. It's easy to run into a situation where the Win32 subsystem won't play nicely with you, because it just doesn't have enough information.

How is that weird?

A semi-hidden API like this is a huge benefit to Microsoft itself, and a detriment to its competitors, or indeed, anyone who writes software that becomes a DLL in the next version of Windows.

If you write to a documented API (Win32, GINA, etc) your code is slower and possibly less capable than it could be.

If you write to the undocumented NT-native-API, you have to invest time and effort into understanding it, and risk having it change out from under you at the next rev.

Microsoft internally, and/or favored partners, get to use the faster, capable native API.

So, Windows developers have something of a dilemma to deal with, don't they?

Also, I'm given to understand they're callable as C functions.

Shared resources across multiple concurrent processes is not a good way to ensure a full night's sleep. I would follow the documentation's advice and pass arguments to the child process and not try and depend on the state of global objects.

Agreed. I think the #1 rule of multi-platform support is to NOT support multiple platforms. Been there, done that, you won't like the tee-shirt.

But if you "must" do it, let the implementation/agent for each platform be as different as they need to be, with no shared mutable memory, and just pass messages.

Reply on news.ycombinator.com