BSD kqueue is a mountain of technical debt (2021)

I came to the comments hoping to get more explanation. I was waiting for the full explanation of the technical debt but then the article just came to an abrupt end.

I wonder if the article started as a short history and explanation of differences, but then the dramatic headline was chosen to drive more clicks? The article could have been interesting by itself without setting up the “mountain of technical debt” claim in the headline.

Synchronous I/O looks appealing for beginners and simple projects. It seems like the simplest path for things where I/O isn’t the primary focus.

Then one suddenly finds themselves juggling ephemeral threads to “background” tasks or such. The application quickly becomes a multi-threaded monster and still suffers responsiveness issues from synchronous use.

The Tower of Babel created from third party runtimes further adds to the mess even for those who try to do better.

Broadly speaking it is difficult to use async I/O well without understanding the implementation details. It is a sharp tool for experts, not magic “go faster” juice.

This means that you neither epoll nor kqueue nor poll work if you ever need to read from a file.

I wish the BSDs and Linux and other platforms would align on non-blocking IO semantics. I just hope it's something new which can handle non-blocking IO with files too.

There also is the thing that, sometimes, different types of event are identical types, lower down in the stack. A simple example are keyboard and mouse events generated from USB devices.

Why would the low-level USB stack have to do a switch on device type to figure out what subsystem to post an event to?

Think about async programming you have a single `await` keyword that can await lots of different things.

One way to think about this is whether all these non-file fds are useful for a variety of generic operations, or whether they're only used for epoll and one-off system calls or ioctls specific to that type of fd. If it's the latter, it seems hard to believe that there actually is some kind of composability advantage.

So, what can you do with them?

1. You can use these fds with poll()/select(), not just epoll. That's not a big deal, since nobody really should be using either of them. But it does also mean that you should ideally be able to use them with any future event notification syscalls as well. And the invention of new mechanisms is not hypothetical, since Linux added io_uring. I'd be curious to know how well io_uring supports arbitrary fd types, and whether they got that support "for free" or if io_uring needs to be extended for every new fd type.

2. You can typically read() structured data from those fds describing the specific event. With kqueue all that data would need to be passed through struct kevent, which needs to be fixed size and generic enough to be shared between all the different event types.

3. You can pass individual fds between processes, either via UDS or fork(). I expect you would not be able to do that for individual kqueue filters.

4. You can close() the fd to release the underlying kernel resource. This doesn't seem interesting, just listing it for completeness.

So there's probably enough smoke there that one could at least argue the case. It's too bad the author didn't.

It's absolutely true that within the bounds of what the OS supports, both interfaces are complete. But that's sort of a specious point. One requires a giant tree of extra types be amended every time something changes, while the other exploits a nearly-five-decade old abstraction that already encapsulates the idea of "something on which you might want to wait".

However, this is still subject to the problems you mention, as well as that you presumably can no longer control events for a file descriptor if you do longer have it, so it is still seems a problem.

Putting the file descriptor in the "struct epoll_event" instead of "epoll_data_t" would have avoided the problem of events on the wrong file descriptor, but that still might not be good enough. (It could specify -1 as the file descriptor for events of file descriptors that you do not have access to.)

Some of this is just the problem with POSIX in general. I did consider such problems (of file descriptors and of event handling) in my own ideas of operating system design, which uses capabilities. I would want to avoid the mess that other systems is doing, too. A capability in this case is fundamentally a different data type than numbers (and IPC treats them differently), although I am not sure yet how to handle this in a suitable way within a process (tagged memory might do, although not all computers use tagged memory). (Note that most I/O is done using IPC and not by system calls; the number of system calls is limited and is usually only used for managing the capabilities and event handling. This would also improve security as well as other things.)

Genuinely would be fascinated to understand what that means

Learning from the past can indeed lead to better designs.

> Hope security issues with it are solved

So, if I understand that correctly, it also introduced new problems? If so, do we know those new issues are solvable without inventing yet another API?

If not, is it really better or just having a different set of issues?

> While I agree that epoll doesn’t come close, I think that’s actually a feature that has lead to a much more flexible and composable design.

You could read it as "kqueue is more prone to tech debt than epoll if events keep getting added".

When adding a new descriptor type, one can define semantics for existing filters (e.g., EVFILT_READ) as one pleases.

To give an example, FreeBSD has EVFILT_PROCDESC to watch for events on process descriptors, which are basically analogous to pidfds. Right now, using that filter kevent() can tell you that the process referenced by a procdesc has exited. That could have been defined using the EVFILT_READ filter instead of or in addition to adding EVFILT_PROCDESC. There was no specific need to introduce EVFILT_PROCDESC, except that the implementor presumably wanted to leave space to add additional event types, and it seemed cleaner to introduce a new EVFILT_PROCDESC filter. Process descriptors don't implement EVFILT_READ today, but there's no reason they can't.

So if one wants to define a new event type using kevent(), one has the option of adding a new definition (new filter, new note type for an existing filter, etc.), or adding a new type of file descriptor which implements EVFILT_READ and other "standard" filters. kqueue doesn't really constrain you either way.

In FreeBSD, most of the filter types correspond to non-fd-based events. But nothing stops one from adding new fd types for similar purposes. For instance, we have both EVFILT_TIMER (a non-fd event filter) and timerfd (which implements EVFILT_READ and in particular didn't need a new filter). Both are roughly equivalent; the latter behaves more like a regular file descriptor from kqueue's perspective, which might be better, but it'd be nice to see an example illustrating how.

One could argue that the simultaneous existence of timerfds and EVFILT_TIMER is technical debt, but that's not really kqueue's fault. EVFILT_TIMER has been around forever, and timerfd was added to improve Linux compatibility.

So, I think the article is misguided. In particular, the claim that "any time you want kqueue to do something new, you have to add a new type of event filter" is just wrong. I'm not arguing that there isn't technical debt here, but it's not really because of kqueue's design.

The event filters must have a standard interface, so adding one more kind should not matter for kqueue more than the existence of a new kind of "file" handle matters for epoll.

The epoll API makes opaque for the user the differences between the kinds of kernel handles, but at some point inside the kernel the handles must be disambiguated and appropriate actions for each kind of handle must be performed, exactly like in kqueue various kinds of event filters must be invoked.

Undoubtedly, I'm not sure why technical debt is in the title

> this is vapid clickbait

This seems unfair. It is a well written discussion of the difference between kqueue on BSDs and epoll on linux, and a historical overview of predecessor APIs. It just has nothing to do with technical debt, which, given the title, is admittedly odd

Unfortunately, sychronous signals of some form are here to stay. That is, having that interrupt-like context switch blocking the same thread. But the ergonomics could definitely be improved. Seems like exception ports are quite useful, but to my understanding they lack some flexibility and performance, so in the end I'd like to emphasize the interrupt style as the bare minimum. Self-paging in Nemesis and Barrelfish reflects this "delegate an individual's responsibility to the individual" principle. Definitely still use a message queue abstraction if applicable, I love that stuff. And since I want the whole state-machine-async-event-loop hog anyways, a top-level event loop fits in nicely as the handler for synchronous signals.

> In fact, these pidfds are not even like traditional Unix open file descriptions, whose rights are based on the credential of who opened them. This makes privilege-separation architectures impossible with pidfds, but I digress.

I knew about capabilities and file descriptors but not this about pidfd...that is an ouch indeed.