Introducing Sd-Event: The Event Loop API of Libsystemd | Hacker News Reader

Introducing Sd-Event: The Event Loop API of Libsystemd

0pointer.net

56 pointsmaxmouchet51 comments

sd-event was announced to be the next systemd-private interface turned public after sd-bus a while ago, and here we are.

If anything, it only further convinces me what a leaky abstraction epoll(7) is, plus all the myriad of *fd calls that really just canonize things like the self-pipe trick in the completely wrong layers. I'd probably use something like libkqueue if I wanted a minimal event loop.

Agreed -- implementing epoll(7) for LX-branded zones on SmartOS[1] was a bit of a shock in terms of how unbelievably broken the abstraction is. I had assumed (ha!) that epoll(7) would have been inspired by the prior art: I/O completion ports on Windows, kqueue on the BSDs, and event ports on Solaris. Instead, despite being the last to the party, epoll(7) is but a more efficient variant of poll(2) -- retaining the broken-by-design aspects of poll(2) with respect to multi-threaded programs. (For details on these conditions, consult the strange Q&A in the epoll(7) man page[2] or the aborted attempt to rectify it with EPOLL_CTL_DISABLE[3].) I suppose that sd-event may represent an improvement in that it attempts to patch some of the leaks in the abstraction -- but I (certainly) agree that kqueue represents a much better design, and to the degree that libkqueue is able to provide that abstraction on Linux, a much better alternative.

[1] http://www.slideshare.net/bcantrill/illumos-lx

[2] http://man7.org/linux/man-pages/man7/epoll.7.html

[3] https://lwn.net/Articles/520198/

Your BSD Now interview[1] where you talk about epoll/kthread is informative and entertaining. The whole interview is worth watching.

1) http://www.bsdnow.tv/episodes/2015_08_19-ubuntu_slaughters_k... around 55:59

The story about restricting rm -rf / without violating the letter of the standard is also great stuff.

That's probably a good question to ask about any piece of software or standard: "Be honest, Is there a way that your software or software written to the specification could render the computer unusable for someone not trying to render the computer unusable?"

A major problem with sd-event as an epoll replacement lies in how it handles multi-threading: It doesn't.

The most error-prone part of using epoll in a multi-threaded environment is free-ing of the per-descriptor state. You never quite know when the last epoll_wait that still knew about that descriptor has fired. There are multiple approaches for dealing with this problem, including RCU-like waiting for each thread to pass epoll_wait once, recycling descriptor states by including an anti-race generation counter in the unused bits of the event user data, or garbage collection.

In contrast, with IOCP or other completion-based programming models, you can reference count the active I/O operations and free once the count reaches zero.

Do you, or anyone else, have any experience from libkqueue?

After a quick glance I'm not convinced that it is battle-tested enough, and the Android port is not building, Solaris port not really working, etc.

kqueue and kevent themselves are available across all of the BSDs; in FreeBSD, OpenBSD, NetBSD, and their various derivatives including Darwin and others.

* https://www.freebsd.org/cgi/man.cgi?query=kevent

* http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/...

* http://man.netbsd.org/HEAD/usr/share/man/html2/kevent.html

* https://developer.apple.com/library/mac/documentation/Darwin...

This most definitely has been "battle tested" and it long since has provided a means for programs to use a single mechanism to wait for I/O events, directory modifications, signals, child processes, and timers. And it has some interesting abilities, too, such as the ability to keep track of grand-child processes forked off by a child, which one can use to good advantage to handle poorly written daemons that erroneously fork-child-then-exit-parent.

libkqueue should not be conflated with this. It provides a workalike API. But there are differences. Some of these are of necessity. Because of the way that Linux signalfds work, for starters, it has been quietly re-specified (the alteration to the wording of the manual page being a subtle one that isn't marked in any way) so that EVFILT_SIGNAL code has to be written differently when using libkqueue on Linux to when using real kevent/kqueue on a BSD. Others are accidents of implementation. One can get spurious wakeups from libkqueue that one doesn't get from real BSD kevent().

Its Linux portion also contains a number of bugs. Internal unexposed file descriptors, for all of the signalfds, timerfds, epolls, inotifys, and whatnot that it opens, are not marked close-on-exec, meaning that there's a significant security concern in any program that uses libkqueue in a privileged process and then forks and executes unprivileged children. The implementation of EVFILT_VNODE/NOTE_WRITE for directories is simply wrong. There are pitfalls when using EVFILT_READ with terminal devices. And its inotify usage doesn't account for the fact that notifications are not always the same number of bytes long.

The first two of those bugs are fairly easy to fix. I never got to the bottom of the third. The fourth is more complex, but is addressable I think. The main problem is not fixing these bugs, but getting those fixes available across a multiplicity of Linux distributions, some of which are operating from years-old versions of the library. So even though I wanted (and still want as a matter of fact) to get libkqueue made better, I had to go with dropping libkqueue and implementing my own kqueue()/kevent() workalikes for Linux.

* https://www.mail-archive.com/supervision@list.skarnet.org/ms...

> it only further convinces me what a leaky abstraction epoll(7) is

epoll isn't an abstraction at all. It was never designed to be. epoll was designed as a far more scalable replacement for poll and select.

That's a very baffling thing to say. Just what is a file descriptor to begin with but an abstraction? And what is a mechanism to multiplex on file descriptors other than an abstraction?

epoll isn't designed to hide anything from you. If anything, it's designed to expose more of the low-level details to allow building a more scalable event loop. For instance, see the whole edge-triggered versus level-triggered bit, a distinction wholly unavailable in poll or select.

In particular, epoll wasn't designed to be a "higher-level" event loop than poll or select; while more powerful, it's arguably lower-level. Hence my comment that it isn't an abstraction.

I am confused about the fact that this is "something that allows us access to all the Linux-specific bits, instead of limiting itself to the least common denominator of UNIX." Doesn't that make anything written with this incompatible with, say, FreeBSD or MacOS?

It does. He's saying that instead of writing a library that abstracts using the least common denominator across all kernels, it only cares about Linux. Doing this allows you to make use of Linux-specific APIs (for example cgroups elsewhere in systemd).

> abstract

Leaking the lower-level interface is a terrible "abstraction".

> least common denominator across all kernel

That's one approach. A better approach is to stop leaking the lower level interfaces and providing the higher-level API you want in a way that isolates you from the details.

When porting to another platform, this may require re-implementing features that are not available on the new platform. This is one of the reasons porting can be difficult.

> allows you to make use of Linux-specific APIs

We never had a problem using those features in the past. This is one of the reasons autotools/cmake exists.

You can't generally re-implement features of a kernel on another platform.

You usually can in any situation where you would be porting the library in question. I'm going to have a hard time if I want to make OpenGL calls work on my old Z80 micros... but why would I want to?

Anywhere I would want to port a library that uses epoll it is trivial to re-implement the functionality using poll(2)/select(2) or even simply setting O_NONBLOCK and busy-waiting while manually polling read(2). Other platforms may have other non-standard features that are full or partial replacements. Performance may not be the same, but that's a common problem when porting software.

I'm sure it's possible to find various pathological cases where this is difficult. In the common cases, proper abstraction is worth it simply for the complexity and maintenance costs.

edit:

None of this should be taken as a recommendation for the epoll interface. vezzy-fnord is right - there are much better solutions available.

Sure you can. It might be the case that the level of granularity is low enough that you'd need to do it in kernel/supervisor mode itself to access private data structures like the process control block, instead of doing compatibility layers, shims or daemons in user space.

In most cases, the latter is fully feasible. You just generalize by categorizing the functionality of a source kernel and finding the equivalent taxonomy in a target kernel. It's not like any commodity OS kernel these days doesn't have a feature to multiplex I/O on ports, fds, handles or whatever that it's that prohibitive to shim a syscall, or more often than not, substitute it at build-time or just use an existing library/framework that encapsulates the details.

> It might be the case that the level of granularity is low enough that you'd need to do it in kernel/supervisor mode itself to access private data structures like the process control block, instead of doing compatibility layers, shims or daemons in user space.

Not all kernels allow you to do that. Even among those that do, you'd have to have some level of privilege to install it. (I know I'd look very skeptically at any program or library that wanted an out-of-tree kernel module installed.)

And even if you manage to get that code submitted upstream, and they accept it (not at all a given), now you depend on the most recent version of that target OS, and you don't run on older versions of it. Which starts you down the same non-portable road you're complaining about.

That said, I absolutely think making such changes in upstream kernels/OSes is the right approach. Make your kernel/OS better to accommodate your userspace software, rather than making your userspace software worse to accommodate your kernel/OS.

Again, this would be more if you needed to emulate things like precise kernel thread semantics, page eviction strategies or whatnot. When talking about system calls (the overwhelming majority of cases), then those are usually well-defined, relatively high level "chunks" of resource management that either have corollaries or can be safely reimplemented in a utility library.

How would you suggest reimplementing cgroups on every other POSIX OS? In particular, the behavior where an entire tree of child processes can't escape from that group? Some kernels may have an equivalent to this (though the semantics may not match), but others don't.

How about prctl's PR_SET_CHILD_SUBREAPER, which makes a process act like PID 1, as the target for reparenting child processes?

Or fanotify/inotify/etc? (A polling-based interface doesn't work here: it misses things due to races, and wastes power by waking up too often.)

Or mount namespaces, network namespaces, and other unshare/clone flags?

You could reimplement those things in another kernel, but almost certainly not in userspace unless a feature with the same semantics already exists in the target kernel.

unless a feature with the same semantics already exists in the target kernel

The point is they do. For instance, Solaris has contracts (direct corollary to cgroups, actually predating cgroups). In Solaris, orphans are adopted by the kernel and init(1M) can be restarted upon a failure condition. Some BSD flavors have procctl(2).

Polling-based interfaces can work, since a port does not have to be completely equivalent in terms of benchmarking, it must conform well by a cost-benefit analysis. File system event notifications are something many platforms reliably have, so that even devices with strict power management policy can incur them (or more often than not they'll incur the cost of polling regardless).

Most Unixes have converged on rfork(2)-like primitives that can customize process contexts to share reasonably well.

I'm not panning systemd for anything, mind you. I'm speaking in general terms. systemd is decidedly unportable, I'm more than well aware and don't have trouble with that aspect.

> In Solaris, orphans are adopted by the kernel and init(1M) can be restarted upon a failure condition. Some BSD flavors have procctl(2).

Neither of which provides the same semantics as PR_SET_CHILD_SUBREAPER.

Unless your desired target platform already has equivalent functionality, "porting" can't happen. It's easy enough to look at individual features and say "Solaris has something kinda like that" or "FreeBSD has something kinda like that". But Linux has quite a bit of functionality with no functional equivalent in other OSes. (And, conversely, some other OSes have functionality with no equivalent in Linux.) And I don't think it's unreasonable to make software that uses the full functionality of a target platform, rather than a less capable subset that can be made to work on other platforms.

> Polling-based interfaces can work, since a port does not have to be completely equivalent in terms of benchmarking,

Race conditions that cause you to miss events aren't OK, though. And even in terms of benchmarking, once people are done complaining about portable software, they complain that it doesn't run well on their platform. And most of the time they don't blame the platform for that.

The point was you don't need subreapers, so you'd presumably just remove the relevant code.

The kernel adopting orphaned processes doesn't make PR_SET_CHILD_SUBREAPER unnecessary. If you just wanted the processes to get reparented somewhere and reaped when they exit, you could let PID 1 do that.

You also seem to have intentionally ignored the general point in favor of picking apart a specific example. The solution to all uses of non-portable interfaces isn't either "you can implement a compatibility layer using features of your target OS" or "you don't really want that code anyway, just rip it out".

Indeed, one can reimplement those things on other systems. And other systems have already done this!

But forcing the use of epoll instead of an existing abstraction like libevent means that now epoll is added to the list of things that must be implemented.

I could go through my code and change all the open(O_CREAT) calls to open(0x200) because that's the same thing. And then none of that code would work on linux, where O_CREAT is 0x40. Would that be an improvement?

> But forcing the use of epoll instead of an existing abstraction like libevent means that now epoll is added to the list of things that must be implemented.

Only if you care about being compatible with Linux. There's a 100% Linux-compatible kernel already available, though. :) And any platform that wants 100% compatibility with Linux could always run it through virtualization.

I'm not saying portability is always a bad thing; far from it. It's also not always a good thing, though. And there's something to be said for the mindset that any time spent on portability is time not spent making the best possible program for the only platform you care about.

Some programs should be portable. It's awesome that Firefox, Libreoffice, and other user-level software runs on Windows, for instance, to introduce people to FOSS and make it easier to switch platforms later.

On the other hand, quite a bit of low-level or server software just doesn't make sense to run elsewhere, especially now that virtualization exists. At some point, it seems like the simplest portability layer is a copy of a Linux kernel and a way to run it on your target platform.

> I could go through my code and change all the open(O_CREAT) calls to open(0x200) because that's the same thing. And then none of that code would work on linux, where O_CREAT is 0x40. Would that be an improvement?

No; that wouldn't be an improvement even for code that only needs to run on one platform, because O_CREAT produces more self-documenting code.

On the other hand, you could write all your code using openat and friends, and never call chdir, which can make software that otherwise had to worry about the working directory much simpler. That may be an improvement, even though that code then won't run on arbitrary POSIX OSes.

openat is posix.

"and friends", most of which aren't.

Oh yes you can, even if it may be ugly; I had to write a porting layer to run 'berkely sockets' networking code on a weird closed source embedded platform that wanted to do all networking through callbacks.

The systemd project doesn't value portability at all and everything they release is basically tied to Linux only

Didn't they do that with cgroups already?

there are many things in systemd tied to Linux-only APIs. This isn't the first, and cgroups is just one example.

I think personally (I may be wrong , please correct me) this was a huge push from Redhat , because they want a Workstation , and with old design (having SysVInit with whole bunch of small and separated stuff) and without launchd clone in Linux world they think they cannot achieve that goal.

Ironically enough, sysvinit and sysv-rc weren't particularly good on the whole "small and separated" aspect [1]. This has been known for a while, and a lot of work has been done to address the problem domain, much of it understudied and not well known [2].

I don't think Red Hat had all that much of a stake initially. Kay Sievers was actually a SUSE employee at the time systemd was conceived for public release, IIRC.

Red Hat's interests seem far more directed at cloud and containers (see: Project Atomic) than workstations, and indeed the direction systemd has been heading reflects that.

[1] http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sy...

[2] http://blog.darknedgy.net/technology/2015/09/05/0/

nonportability is typically a design goal of systemd. See http://0pointer.de/blog/projects/the-biggest-myths.html down at point 16 ("Myth: systemd is not portable for no reason.")

Nonportability isn't a goal, portability though isn't. Portability isn't all that great a goal to begin with. You limit yourself to a lowest common denominator and sometimes you need to make compromises between platforms. This makes it very hard to impossible to do truly innovative and good things.

GUIs and the web in particular are the perfect example. Every GUI that's not specifically designed for a platform is always worse than the native solution. Additionally the web is laughably behind. Trivial WebGL experiments that you could do natively decades ago regularly hit the HN frontpage. On a lower level you have C which is bad for many reasons but one of them is trying to be portable and so you end up with not being able to assume 2's complement.

Whenever you can do without portability, you're probably better off ignoring it.

The web is laughably behind not because of portability concerns, but because it fundamentally began as a poor man's hypertext system that could be hacked up in two weeks for linking together CERN's document archives. Web browsers are an interesting experiment in that they're virtual machines that began from a mostly blank state independent of their host, and progressively had to rediscover decades of CS research (often haphazardly) to get them in their current state.

That said, nothing truly innovative is being done with systemd to begin with.

If you’re willing to restrict the flexibility of your approach, you can almost always do something better.

        — John Carmack

This is yet another leaky abstraction. The non-portability is mentioned:

    If portability is essential to you, sd-event is not your best option.
    sd-event is a wrapper around Linux-specific APIs ...

Also, "allows us to access"? Why the implication that epoll and timerfd were inaccessible?

> Why the implication that epoll and timerfd were inaccessible?

With high level, portable, abstractions, it may very well be impossible to expect the event loop to know how to deal sensibly with, eg, timerfd.

libuv seems to handle such things pretty well. Ported to AIX, *BSD, Darwin, Solaris, and Windows.

Indeed, that paragraph seems to say that they didn't reuse the existing libevent or libev because they were too portable.

Exactly, how to do you embrace, extend, extinguish if you're writing portable code?

The first step is “embrace”. Writing kernel-specific code is hardly “embracing”.

This is the extend step. They've embraced the unix philosophy, and are now trying to make it "better", throwing compatibility with other systems out the window for little reason.

Yes. Their response is "Who cares? We're targeting Linux. If you want BSD, you know where to find it."

Many members of the BSD communities are quite happy with this state of affairs. This is especially true for the BSD users who became BSD users after experiencing severe problems with systemd and Linux. The very last thing they want to see now is systemd expanding beyond Linux distros.

Nope, their response is "BSD is irrelevant. Stop using it. Having only one platform is a reason to party!" [1]

It's quite hostile.

[1] https://lwn.net/Articles/430699/

Duplicates https://news.ycombinator.com/item?id=10601046 .

One more step closer to launchd?

systemd has long been a functional superset of launchd.

Reply on news.ycombinator.com