Comment by pengaru | Hacker News Reader

Comment by pengaru | Hacker News Reader

No new code should be using select(). Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE.

I've personally had to debug and fix crashes due to this assumption at multiple jobs.

Back at Hostway decades ago we had Apache+SSL instances randomly crashing which turned out to be due to something opening /dev/random and using select() on the returned file descriptor in a .so. This only happened when the machine had enough virtual hosts to exceed FD_SETSIZE on the spurious open of /dev/random.

At Bizanga we had random crashes in their "carrier grade mail server" software, which would link in proprietary modules from vendors like Symantec for realtime virus scanning before even accepting messages for delivery. This stuff ran in-process for performance reasons, just in separate threads. Being "carrier grade" meant thousands of concurrent connections -> thousands of open files. Those vendor-provided plugins would phone home behind the scenes for virus db updates. Turned out they were using select() on the socket connected for the update, if there were enough active sessions at the time, the socket's value would exceed FD_SETSIZE and boom: segfault.

People tend to forget that it's the value of the fds you pass to select() which can't exceed FD_SETSIZE, not just the quantity of file descriptors. It could be just one file descriptor, but if it's a high enough value exceeding FD_SETSIZE, it will segfault. The value is used as an index into a fixed size array.

I ended up making an execution wrapper @ Bizanga to launch everything with >FD_SETSIZE file descriptors already opened once I saw the vendor modules were linking select(). We found numerous crashes this way, without waiting for them to occur in production.

> No new code should be using select(). Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE.

Perhaps it might be time to start treating select() like gets() was treated: the compiler should emit a warning by default whenever it's used, and after perhaps a decade, the API should no longer be available for newly compiled code. While not as bad as gets(), which according to Rusty's classification of API quality (https://ozlabs.org/~rusty/index.cgi/tech/2008-03-30.html and https://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html) gets a -10 ("It's impossible to get right"), select() is around a -5 ("Do it right and it will sometimes break at runtime"), or perhaps even a -7 ("The obvious use is wrong").

I agree 100% re: compiler throwing a warning like gets() for select().

This makes me wonder if we could realistically start enforcing SystemCallFilter on new services which immediately crashes an app of it tries to use select(). I suspect there's a lot of them that would fail at the moment... But maybe in a few years.

select() is useful on platforms without any poll variants.

Of which the only one left is Windows, where you probably have a completely different event loop anyway. The Windows select by the way does not have the limitations on the file descriptor that the article talks about, because Windows file descriptors are not small consecutive integers.

…… and at least until 2011 which was when I last checked, select() was the reliable way to do usleep() in Windows without having to change system time granularity, set up invisible windows with their own message pump or a variety of other weird and unreliable ways.

There are plenty of embedded platforms in the same boat.

Actually that’s not totally accurate. The native IO model in the Win32 subsystem, I/O completion ports, is indeed completion based. But the NT kernel does have a scalable readiness model too, limited to sockets, that people have used to build an epoll(). See: https://github.com/piscisaureus/wepoll. This is used by e.g. deno and node.

select() on windows is kinda fine. It uses an actual array in fd_set and you can increase the array length by defining a constant or passing a custom structure.

You might be surprised at how little standards like POSIX actually cover. There are huge gaps in the APIs implemented by e.g. QNX. (I haven't checked if this specifically is one of them but I would be surprised if it isn't.)

> the API should no longer be available for newly compiled code

And break perfectly working code?

Arguably, if it depends on select() then it might unexpectedly break at runtime and is therefore not "perfectly working code".

Add a switch to enable the previous behaviour. If the software is actually maintained than that switch will be added to the build config. If it isn't it shouldn't be used in production going forward. Whoever adds the option to the build process takes responsibility if the bugs it is deprecated to avoid do in fact affect this code.

Maybe I am out of date. (I probably am because I haven't written any code that uses select() for two decades.)

Can somebody please identify the POSIX method by which one can wait on a set of descriptors, that does not involve spinning in a polling loop and gobbling up a CPU thread? The last time I had a need for such a thing, select() was the preferred way to do that.

The article mentions a few near the end, I think.

The article mentions one (poll) and doesn’t make it clear they it’s posix. The rest is Linux specific.

Aside from being a lot more recent (posix 2001), poll was also broken on macOS until 10.9. Though you can always use kqueue instead that remains a portability annoyance: Linux still doesn’t have kqueue.

> still doesn't have kqueue

And with io_uring around it never will unless someone does a pure userspace implementation. Not certain if possible but it might be.

Like: https://github.com/mheily/libkqueue ?

Epoll works really well in Linux.

I'm sure that epoll() works great, but I was asking about a POSIX replacement. Some people still write portable code.

While this is a laudable goal and good for uncovering unexpected bugs, it also means writing for the lowest common baseline. And in this case, the lowest common baseline is unfortunately not very high. poll is what you want, but has its own gnarly gremlins. Better mechanisms are not standardized.

Why not use libevent or libuv, and make it portable for all other platforms?

The POSIX alternative to select() is poll(): https://man7.org/linux/man-pages/man2/poll.2.html#CONFORMING...

See also:

* https://pubs.opengroup.org/onlinepubs/9699919799/functions/p...

As others mentioned, the POSIX alternative is poll() (and ppoll() as the alternative to pselect() ).

If you want both portability and performance (as in, O(1) instead of O(N) as poll() and select() ), there's no portable standard. In that case you're better of with a wrapper library like libuv or livev.

poll() is POSIX but slower than select()

> No new code should be using select(). Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE.

I'm with you here. I'm also guilty of using it still for some CLI tools or other small programs. Just to be clear you don't need to assume all fd values will stay within FD_SETSIZE. Just that the fds you will use with select will stay in that range. Even if there will be lots of fds (more than FD_SETSIZE) for other resources, it works for example to select on stdin if you're not doing funny business reopening it.

There's no much of an excuse to keep using it though other than laziness.

Laziness is pretty good reason to continue to use select. It's a convenient interface for simple tasks. The very nature of fire a set of fds and get a result beats (e)poll/kqueue/fd-registering based API in UX regard. It's sad there is no alternative with same simple interface and without FD_SETSIZE shortcoming.

Edit: I'm biased by Python which has `select.poll` with API similar to epoll and it's really hard to use it instead of select.

poll() is even easier to use, imo.

> People tend to forget that it's the value of the fds you pass to select() which can't exceed FD_SETSIZE

Is that true? I have a memory of working on systems where you can pass larger arrays in if you want, you just have to allocate your own bitvectors instead of using the convenient struct. A quick look at the linux kernel makes me think that this is true for Linux; the kernel doesn't appear to reference FD_SETSIZE in its implementation.

If you read the man page bugs section: https://man7.org/linux/man-pages/man2/select.2.html

"The Linux kernel imposes no fixed limit, but the glibc implementation makes fd_set a fixed-size type, with FD_SETSIZE defined as 1024"

Everyone using select, with very few exceptions, is using glibc.

That limitation does appear to exist in glibc if you look at the source code. The source code requires you to pass a glibc defined 'fd_set' type https://man7.org/linux/man-pages/man2/select.2.html

That struct is defined here: https://github.com/bminor/glibc/blob/595c22ecd8e87a27fd19270...

It does seem to have the limits described by the blogpost.

On FreeBSD's select(2) man page, the NOTES section has the much friendlier

> The default size of FD_SETSIZE is currently 1024. In order to accommodate programs which might potentially use a larger number of open files with select(), it is possible to increase this size by having the program define FD_SETSIZE before the inclusion of any header which includes <sys/types.h>.

Although, if you link to a library which uses select and doesn't check that fds it uses are valid with the FD_SETSIZE it was compiled with, you're still in for a bad time, as suggested in the article. And still, probably use something better than select.

The glibc/musl select library call doesn't verify nfds is < FD_SETSIZE so you could create your own fd_set as large as you want it.

glibc: https://github.com/bminor/glibc/blob/master/sysdeps/unix/sys...

musl: https://git.musl-libc.org/cgit/musl/tree/src/select/select.c

*BSD allows you to define your own FD_SETSIZE before including system headers.

Yep, and if you pass an fd >= 1024, it will corrupt the stack by using FD_SET and your program will crash. I've seen that in production, was solved by forcing glibc to use epoll.

From what I recall (without looking it up, and relying on wetware memory that's 20 years old), there is a set of macros that are used to initialize, add, and remove descriptors to a file descriptor set. FD_SETSIZE defines the maximum number of descriptors that the set can hold. I could be wrong, but I seem to recall that the implementation uses both an array and a bit-mask. The array holds the descriptors and the bit-mask identifies which array elements are "active".

The argument is just an array of unsigned longs, each bit of which represents one file descriptor — there's no magic in there. The add/remove macros just do the typical shift-and-mask style bit twiddling. You can pass a bigger or smaller array than the one in fd_set if you want; TTBOMK the kernel doesn't care. It just looks at the numfds argument to know how large an array you passed in.

Some platforms (but not Linux) let you define FD_SETSIZE before including C headers to override the default. The fd_set structure will be made correspondingly larger. On these platforms it should work fine, though there may still be an undocumented kernel limit.

Some really old platforms with select() don't even define fd_set or FD_SETSIZE, and you're on your own :-)

Linux added the poll() syscall in 2.1.23, and Linux select() had a kernel-side limit of 256 fds up to 2.1.26, when it raised the kernel-side limit to 5120/10752 depending on architecture. It was 2.5 years until the kernel-side limit was removed, by which point everyone would have had poll() in Glibc.

So if you are going to the effort of writing portable select() except using the trick of defining your own, larger arrays, and using poll() when it's available, on Linux don't bother. You'll hit a kernel limit anyway on those very old systems which don't have poll().

> No new code should be using select(). Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE.

What should we use instead? poll()?

that's indeed one of the choices from the article itself.

I remember reading somewhere that poll was just a wrapper for select. I’m guessing that’s either outdated or false now.

It's correct but outdated: according to https://man7.org/linux/man-pages/man2/poll.2.html#VERSIONS "The poll() system call was introduced in Linux 2.1.23. On older kernels that lack this system call, the glibc poll() wrapper function provides emulation using select(2)."

So as long as you use a Linux kernel made this century, you will be fine.

epoll.

Is there a portable alternative to pselect()? There's ppoll(), but it's missing on some platforms.

At this point I believe ppoll has been added to every mainstream Unix except macOS. It's also been accepted for the next POSIX revision.

For many years pselect on macOS didn't actually fix the race condition, notwithstanding its UNIX03 certification. pselect was just a naive wrapper around select that didn't atomically unmask signals. So I ended up having to create a kqueue-based implementation for use on macOS and stragglers like OpenBSD which at that time lacked pselect entirely. See https://github.com/wahern/cqueues/blob/87f14f9/src/cqueues.c... I can't remember if pselect has been fixed on macOS.

The same approach could be used to emulate ppoll on macOS. Though epoll, kqueue, and Solaris ports are so similar that you can easily wrap all of them behind a thin shim. See https://github.com/wahern/cqueues/blob/87f14f9/src/lib/kpoll... (Caution: it's an incomplete refactor of a similar shim I used in some non-open source code. The original code is even simpler and has seen extensive use. The refactored shim should work but is untested.)

AIX is really the only mainstream Unix lacking an API w/ equivalent semantics. AIX has a better-than-poll interface, but it doesn't work anything like the others. Notably, epoll, kqueue, and Solaris ports control descriptors are all pollable in their own right, permitting you to nest different [readiness polling-based] event loop frameworks.

There is a way to simulate it portably, but it's a little complicated. You can use the self-pipe trick portably on POSIX systems I believe. (There is also the sigsetjmp/siglongjmp trick on most platforms, but I remember it breaking on some version of Cygwin.)

The self-pipe trick is where you create a pipe(), add the read side to select(), poll() or other syscall, and have your signal handlers write a byte to the write side. When reading the pipe to flush this state, remember to read until it's empty because there may have been multiple signals. Reading 2 bytes and looping if you get 2 will get this done usually in one syscall.

It's best in principle to make the pipe non-blocking so the writes can never block if the pipe fills. This is basically the portable equivalent to Linux's eventfd. (If you're worried about out-of-memory preventing non-blocking write of a single byte, you can close the pipe instead, but this causes several complications because it's not safe to close the same fd twice, and you may have signals interrupt other signal handlers, so race hazards around closing the fd twice must be protected against by blocking signals inside the signal handler and using more flags.)

You can optimise self-pipe by setting a write-enable flag just before select() etc and clearing it afterwards, with the signal handlers only writing a byte (or closing the pipe) when the flag is set, and setting a separate flag to say that the signal occurred, which is checked between setting the write-enable flag and before the select() etc syscall. Check the signal-happened flag after select() etc and if it's set, read the pipe to flush even if it doesn't show as readable in the fd set, because the signal could have happened later. A further optimisation is for the signal handler to clear write-enable when writing the pipe, so subsequent signals don't do it again. Don't assume this means there can only be one byte in the pipe, because nested signals introduce a race hazard to writing and clearing write-enable atomically.

The sigsetjmp/siglongjmp trick (setjmp/longjmp on platforms without the "sig" versions) is to sigsetjmp() before select(), poll() or other syscall, then set a jump-enable flag (or just jmp_buf pointer) telling your signal handlers to siglongjmp(), then clear the flag after the select() etc syscall returns. This is more complicated than self-pipe in a number of ways (and a bit slower), because you have to be careful to prevent nested signal handlers or deal with them appropriately, and it's doesn't work reliably on all platforms. So I would stick with the self-pipe trick.

> No new code should be using select().

It’s faster than poll, and if you are accept()ing a lot, faster than epoll.

Multiple threads can do better, but only by working together. SO_REUSEPORT+select is faster if you can get away with it. Iouring looks promising though.

> Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE.

That’s what the rlimit is for. If you call a program that uses select with 1025 open FD you get what you deserve.

>> Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE. > >That’s what the rlimit is for. If you call a program that uses select with 1025 open FD you get what you deserve.

I thought that FD_SETSIZE was a compile-time constant that limited what values for file descriptors could be passed to select.

IOW, it doesn't matter if you have only two files open; if one of them has an fd with a value of 1026, then select will do the wrong thing.

Yeah but fds are guaranteed to be opened lowest available on first, so thats ok, you cant get fd=1026 without having 1026 open.

The Linux syscall does not have a fixed FD_SETSIZE so glibc could increase the size.

> Yeah but fds are guaranteed to be opened lowest available on first, so thats ok, you cant get fd=1026 without having 1026 open.

This is exactly correct.

> The Linux syscall does not have a fixed FD_SETSIZE so glibc could increase the size.

The application writer can define FD_SETSIZE before including <sys/socket.h> and set it to whatever they want on BSD (including OSX). glibc could* adopt this trick, but my feeling is you probably don't want to stack-allocate this stuff anyway.

Dealing with a large number of very-active connections concurrently efficiently requires more effort than just this though.

To be more precise on one bit, you can't get fd=1026 without having had at least 1027 open at some point in the past. You can subsequently close a bunch of them in the middle.

A compatibility hack for some applications reliant on badly-done select() might be to set RLIMIT_NOFILE to FD_SETSIZE.

> To be more precise on one bit, you can't get fd=1026 without having had at least 1027 open at some point in the past.

Of course you can.

    dup2(0,1026)

Hah! Yes, okay. I should have said “… by opening”.

> Iouring looks promising though

It's more than promising. It's an unbridled joy to work with (compared to things like epoll) and performs very well already.

I am very pleased to hear that, and I'm looking forward to trying something daring with it.

> No new code should be using select(). Using it assumes the execution environment keeps all file descriptor values within FD_SETSIZE.

That's what most of the article is about.

n.b. FD_SETSIZE on Windows represents the quantity (!)

Reply on news.ycombinator.com