Subprocess: Don't close all file descriptors by default (close_fds=False) | Hacker News Reader

Subprocess: Don't close all file descriptors by default (close_fds=False)

bugs.python.org

35 pointsluu22 comments

Meanwhile VSCode's terminal still leaks Electron fds: https://github.com/microsoft/node-pty/issues/657

I get this proposal's rationale, but it seems that it would implicitly make fd leaks more prone in python programs

Of all the quirks with process spawning in posix keeping file descriptors open is the most problematic one I encountered. This bit into my ass while implementing a C library to have proper process spawning and stdio handling in LUA. I really wish file descriptors were non inheritable by default.

> I really wish file descriptors were non inheritable by default.

In Python 3.4, they are (released ten years ago).

But in POSIX, they are not, so any module implemented in C is still potentially problematic.

Only if that C-implemented module uses raw C to create file descriptors. And if the module has not gotten an update in the past ten years to fix the problem.

This can help: https://man7.org/linux/man-pages/man2/close_range.2.html

    close_range(3, ~0U, CLOSE_RANGE_UNSHARE);
    execve(....);

...but it's not very portable yet.

I’ve never heard of it before. I work in an embedded environment where kernel version is known beforehand, so portability won’t be an issue. Thanks.

The counter-arguments presented seem persuasive. This was originally submitted in 2020, and closed in 2022. Why is it relevant or interesting today?

https://github.com/python/cpython/issues/86904

(The comments before 2022 are all the same; the newest comment is only on Github.)

Go goes slightly further and opens all descriptors with O_CLOEXEC by default, so if you ever execute an external command you have to go out of your way to preserve any descriptors, which is really nice in my opinion

Python has done the same thing for the past 10 years for fds created by the runtime (Python 3.4). But 3rd party extensions / modules may create non-CLOEXEC fds.

FDs really should have been opt-in for inheritability from the start, with the possible exception of stdio. Inheritability being an fcntl is definitely one of the worst bits -- if the APIs for fork()/etc were designed today it would probably take a list of FDs that would be dup2'd in the new child process.

FDs optin, memory too given the insane performance pitfalls architectural issues and pernicious security problems (think cloning a CSPRNGs internal state) and suddenly you realize CreateProcessA was and always has been the superior API.

> suddenly you realize CreateProcessA was and always has been the superior API

Throw in IOCP and you realize that NT is a pretty solid, well-thought-out OS

Speaking as a linux addict from the 90s: The NT kernel was great. The windows GDI and USER were reasonable but a bit warty, and not fully up to date for current graphics works. COM as interoperability protocol was quite good. The windows UI from 95 to win2k was also great.

Really apart from the lack of unix shell like automation and some serious security holes, this was a good system. Unfortunately, starting with XP, the end user was not Microsoft's center of attention anymore, and the OS gets worse every new version since.

> FDs really should have been opt-in for inheritability from the start, with the possible exception of stdio.

Yes.

> Inheritability being an fcntl is definitely one of the worst bits

Well, or you can use O_CLOEXEC in most APIs that create an fd.

Yep, totally, although there are still some holes in that. pipe2 being missing on Darwin is one example. You also need to hope that all the libraries you're integrating with are using O_CLOEXEC as well.

Even posix_spawn didn't fix this problem. https://pubs.opengroup.org/onlinepubs/007904975/functions/po...

(2020), or perhaps 2021.

> On macOS, posix_spawn() is even a syscall.

First time I hear about this, interesting. I wonder what the performance benefits are like.

Dang: Link should be https://github.com/python/cpython/issues/86904

Reply on news.ycombinator.com