Linux Programming: Signals the easy way
stev.org
stev.org
http://www.boost.org/doc/libs/1_64_0/doc/html/boost_asio/ref...
I'm not sure it's correct but it might work here because the only place it's where it's read from without locking is the main thread (`while(cond->running)`), which is also the only place where it's written to (with locks this time).
I'd err on the side of caution here and go with the rule of thumb "volatile is almost always wrong". The slightest change of the code (e.g. allowing another thread to terminate the app) will cause it to fail. It's not much effort to change it to __atomic_load/__atomic_store and remove the volatile qualifier or lock/unlock the mutex every time it's being accessed (costs about 50 nanoseconds).
In the specific example given in the article, not only volatile, but even _Atomic would be completely superfluous. 'running' it is set once in main before any threads is spawned so there is no concurrency issues; after that every read and write access is done under the protection of the 'lock' mutex, so there is no need for any special qualification in POSIX or C11.
Interestingly, a 'volatile sig_atomic_t' used in a signal handler is both required and sufficient to communicate with the thread of execution interrupted by a signal handler and is the only portable use of the keyword.
edit: there is another portable use in conjunction with {sig,}{set,long}jmp of course.
The first 'running = 1' is not guaranteed to be visible to other threads because there is no synchronization around it. In practice the pthread_create will probably "fix" the race condition.
My best understanding is that you need a mutex or an atomic every time you access shared data to guarantee visibility. Most CPU architectures don't need the implicit memory barriers from mutex or atomic but it's not wise to rely on it.
This is formally guaranteed by C++11 and C11 for std::thread and thrd_create, and also by posix for pthread_create although the language is more informal.
edit: missing step
But it makes sense, it would be more difficult to write correct code without it.
In fact it looks like geofft is in this thread below :)
>So my claim that there’s a better alternative to signalfd is wrong, but it’s still true that signalfd doesn’t solve any of the other troubles with signal handling: it just saves you a separate thread.
Of course this sentence is disingenuous. Avoiding a separate thread is a huge gain. Dealing with concurrency is one of the main problems with signals. So yes, signalfd is absolutely hugely useful, allows for a lot of simplification, and should be the first tool you reach for if you want to handle signals on Linux and don't need portability.
But as far as I know, the only reason you'd ever care to actually count individual signal delivery events is with SIGCHLD. So when waiting for SIGCHLD, using signalfd is not sufficient. You'd need to also use one of the wait/waitid/waitpid variants to ensure that you can respond appropriately to each individual SIGCHLD delivered.
What do you do in that situation? You can force back pressure semantics on senders but then those can grind down, a lot of classic Unix tools were never built with that case in mind.
You could even argue that for signals coalescing should be preferred behavior and multiple signals could be merged together SIGIO, SIGCHLD. In SIGIO you can merge signals impacting FD, with SIGCHLD you should already be running waitpid with WNOHANG.
Sadly I don't believe this can uniformly apply to all signals (don't want to coalesce SIGSEGV/SIGBUS). Also siginfo_t complicates things quite a bit (since the info in there makes it hard to dedup). But such is life in POSIX.... we only now have the hindsight.
Can signalfd be fixed? I think it can, if we explicitly tie
signalfd into the signal-disposition mechanism. If signalfd
could claim responsibility for signal delivery, instead of
requiring that signals be masked or ignored in addition to
using signalfd, this would solve both problems.
Basically all Linux needs to do is copy BSD. But that will _never_ happen, because that would be too sane.
There's no way to fix the issue with coalescing. The kernel cannot buffer an unlimited number of events--memory is finite. If your process doesn't loop on wait4 it's fundamentally broken.
The pipe trick doesn't help because you can overflow the pipe buffer, too. On a large, heavily loaded system, receiving several thousand signals before any can be processed could easily happen. In the context of a mechanism like signals where there's no way to throttle the sender with back-pressure, coalescing is the best and correct answer.
The same issue pops up with inotify, which specifies the IN_Q_OVERFLOW event for the same reason. The kernel simply can't buffer an unlimited number of events. Nor can it block file operations when the buffer is full. It can't coalesce events, either, because they could all be from different files, and in any event I think inotify tries to preserve ordering.
Unsurprisingly, with EVFILT_VNODE kqueue implements file notification events only for open file descriptors. That is, you can only listen for events on files for which you have an open reference. File events are then coalesced into a buffer associated with this open reference. Events can never be lost, though ordering can be lost. Which means that compared to inotify, it's much easier to write _correct_ software that doesn't accidentally fail because you forgot to handle the rare overflow scenario, or handled it incorrectly.
Of course, inotify is much more convenient than EVFILT_VNODE. It also uses fewer resources for the common case. Which is a classic distinction between Linux and everybody else: Linux makes the common things blazingly fast and easy but makes it difficult to write resilient software, or to implement novel solutions not conceived of originally.
Partially false: SIGABRT can be caught. Indeed, that's one of the methods used by the abort() function to stop: https://github.com/lattera/glibc/blob/master/stdlib/abort.c
we must have been looking at the same codebase...
for posterity, the correct solution is to use a posix semaphore which is async signal safe.
I've considered designing some sort of complicated DSL in Rust to construct async-signal-safe handlers, but it didn't seem worth the complexity. (Although, now that I write this comment out, I think you can handle SIGSEGV safely from another thread as long as your first thread calls for help in an async-signal-safe way, so maybe that would be a library worth writing.)
For the rest, using sigwaitinfo or signalfd works just as well in Rust as in C/C++, but not really better or worse. There aren't any good libraries for it (which I keep meaning to do something about), but the underlying POSIX APIs are usable equally well in all three languages.
As for whether you should worry about it? Try not to use signals, regardless of language. If you're writing some sort of demand-paging system using SIGSEGV handlers, I hope you know what you're doing already. If you're thinking about SIGALRM, there are other ways to get a notification after a fixed time (and Rust's mio/tokio, for instance, gets you timers in your event loop). If you're trying to use SIGUSR1/SIGUSR2, please rethink your design - you'll be much happier with something like a UNIX socket listening for connections or a localhost-only URL for debugging or whatever. If you want to handle SIGINT (intercept Ctrl-C at a terminal), SIGQUIT (intercept Ctrl-\ at a terminal), SIGTSTP (intercept Ctrl-Z at a terminal), SIGTERM (intercept the "kill" command's default signal), SIGHUP (do something when the terminal exits), SIGWINCH (get notified when the terminal resizes), or SIGCHLD (get notified when a child process exits), you do have to use signals, but all of these can be run through sigwaitinfo or signalfd. And you don't have to actively worry about them if you don't want to customize their behavior, or if (for SIGINT, SIGHUP, etc.) you just want to ignore them and not do anything special.
The only odd case is SIGPIPE, and Rust's standard library blocks SIGPIPE for you and returns EPIPE from a system call instead, and unblocks it in child processes, so you don't need to worry about it. (The default SIGPIPE behavior is basically legacy compatibility with existing C code from the very early days of UNIX.)
edit: I should have read ben0x539 comment which suggests the same thing.
Yes, you could write a library for use inside signal handlers that shares no state with the actual libc. Then you could also safely printf, for instance, although you'd jump ahead of any printfs in the host program that are still buffered, etc.
But it's not just libc's globals that are a problem, it's your own globals and those of any other libraries, and if you intend to do nontrivial work in the signal handler (such that having a compiler check that you're async-signal-safe is worth doing), you probably are actually trying to share state with the main program. For instance, say you're installing SEGV handler to virtually map a large data file into memory, fetching it from some remote file storage API as needed. You want access to some data structures from the main program to figure out what to map, probably the ability to make HTTPS connections, etc. And you have to do that all from the signal handler, because whatever pointer segfaulted has to be working by the time you return from your signal handler.
You might be able to do this with Rust plus cleverness, by putting the signal handler in a separate statically-linked crate and using the ownership system to track the fact that the signal handler needs read-only access to some data. (Or with dlmopen, in C.) But it would still be very complex.
There are a couple crates which should suffice for basic SIGTERM/SIGHUP/SIGINT handling:
https://github.com/BurntSushi/chan-signal creates a sigwait thread which sends signals over a channel.
https://github.com/alexcrichton/tokio-signal uses the self-pipe trick to integrate with a tokio event loop.
How easy is it to fuck up the signalfd thing and end up not responding to SIGINT in a timely manner, because I'm being silly and doing some blocking work in the signalfd loop for some reason? Does that happen to people at all?
What kinda usecases are there for handling SIGSEGV? I'm not the kind of person to write on-demand-paging systems so I'm not really sure what that involves and when I need that sort of thing.
If you're building a runtime that includes go like goroutines, you could handle SIGSEGV when you run out of stack space and graceful kill the go routine.
You can use it for some kind of online migration of VMS.
You could build a moving garbage collector system for native code multi-threaded code where you don't need to stop the world... you can deal with race of not stoping / moving by making those pages in accessible and only stoping the thread / fixing up if happens to hit moved memory.
A runtime for a language that has multi-threading system that doesn't share VM mappings (which is possible using the clone sys call on linux) but can share other things.
All somewhat advanced uses. But uses never the less.
Implementing a VM for Java or similar in which you need to trap null pointer accesses and turn them into exceptions is one use case.
SIGBUS is simialr to SIGSEGV (It happens if a mmap is larger that the backing file and you try to read the unbacked memory). You basically have to handle it if you don't control the backing file, Eg: with shared memory when the other process resizes it after you've mmap'd it. libwayland-server can install one for you that replaces the map with 0'd memory.
Another use case would be a massive one to one hash table. You allocate the address space and when you fault on it you load data from a database on demand and continue execution.
https://www.kernel.org/doc/Documentation/vm/userfaultfd.txt
There's also something in Mach for this on OS X, but I forget the details. And a handful of non-UNIX OSes have had support for an outside program being able to get notified efficiently on segfaults and map pages in.
another one: attempt a best-effort flush of your log queue/buffer.
If you care deeply about reliable logging in the presence of arbitrary segfaults, get it to a buffer outside of the program: either write(2) / send(2) it to a file/pipe/socket, or write it to a shared memory region / mmaped file and pay attention to the atomicity and ordering of memory writes on that region.
You might also conclude that arbitrary segfaults aren't in scope, and there's some specific high-risk code that is separate from your logging implementation. If you can prevent wild pointers in the high-risk code from reaching the pages used by the logging implementation (e.g., you're a language VM implementation or something), you can relax your requirements. In particular, if you can also prevent them from reaching the libc, and you know the high-risk code cannot induce a segfault in the libc, then you can probably start calling async-signal-unsafe functions. But this requires careful thought.
On the logging system I'm familiar with, the logging queue is lock-free, so any invariant is preserved at every instruction boundary. The log is actually written to disk from a background thread, and the signal handler is responsible of signaling the log thread to flush the logging queue(s).
Of course if the log thread itself has generated the sigsegv (because a log entry or the queue itself is corrupted by a bug) you are out of luck, but you are no worse than not attempting the flush.
Agree that making the log thread a separate process instead is significantly more robust (and not significantly harder in fact).
Yes, but if you're doing nontrivial work, you probably don't just want memory for yourself, you want to share some actual program state with the main process. See the other comment I posted.
In practice, making any raw system call is async-signal-sae. POSIX doesn't specify this because it doesn't specify what is a system call at all - it's purely a C-language, compile-time interface, and a unikernel that makes everything a library call, a Windows subsystem, etc. are all valid ways to implement it. But on a system where mmap is in fact a system call (and the libc wrapper does nothing interesting) it follows from first principles: there's no userspace resource that can be left in an inconsistent state.
> How easy is it to fuck up the signalfd thing and end up not responding to SIGINT in a timely manner, because I'm being silly and doing some blocking work in the signalfd loop for some reason? Does that happen to people at all?
I mean, this is basically the same question as "How easy is it for a system that's only accessible via an HTTP API to get stuck processing an event and not process a POST /quit". If you're worried about messing up your event loop and getting stuck, then yes, SIGINT interrupting your process is useful -- but also, you've already admitted you don't know what the state of your process is. So an orderly shutdown is difficult. Any work you could do to make it easier turns out to be work that makes your original non-blocking design get implemented correctly.
You should probably just leave SIGINT at its default disposition (terminate the process immediately, same as SIGKILL) and save state to disk periodically or something. Or, at best, implement a signal handler that does very little interaction with process state, maybe just writing a log message (but note that syslog isn't async-signal-safe!) and exiting.
Regarding Rust specifically, it would seem signal handling remains an open item[1] though some degree of support appears present.
Huh?
If a thread dereferences a null pointer, the sigsegv is only relevant to the thread that dereffed the NULL pointer - that's the only thing that can do anything with it. It makes no sense to send that signal to another thread - it's a signal generated synchronously by the thread itself, rather than asynchronously from anywhere else.
> Some signals will be specific to threads anyway.
I.e. some signals are not sent to the process but to specifc threads. See 'bleep' reply for details.