Runit – a Unix init scheme with service supervision
smarden.org
smarden.org
How does it address the "who-watches-the-watcher" problem typically associated with service restarting?
There are many reasons a process can die that are outside of its control, including signals from outside the process, handled (but uncorrectable) memory errors, and the OOM killer (on Linux). Besides that, it seems like a major design shortcoming if fatal errors in any particular program (however critical and however simple that program may be) can be unrecoverable for the whole system.
It's definitely possible to solve this problem rigorously and completely, though I don't know of a way to do it without support from the kernel. On illumos systems, the service restarter ("svc.startd") provides a complex restart policy for user-defined services. I believe the restarter itself is restarted blindly by init, and init is restarted blindly by the kernel. If the kernel dies, the whole system is rebooted. In this way, if any software component in the chain of restarters fails, the system still converges to the correct state.
At some point, you just have to assume that some critical components are working correctly. Adding complexity just makes it harder to reason about it, or, depending on how paranoid you are, prove it.
Agreed on minimizing complexity. The only part of the chain I described that's very complex is svc.startd, and that's largely to support rich configuration.
Also don't mistake my position for saying that quality isn't important. Rather, just that perfection is not a reasonable constraint.
If the svscan process dies, then your system is still chugging along and you can intervene to restore the supervision tree (otherwise svscan inspects supervise processes at a regular 5s interval). If you have some really critical process, then you could integrate a checkpointer into the run script chain so that you can just pick off from the last image of the process state with minimal interruption.
At some point, for some component or set of components, perfection is your only choice, regardless of the rest of your design. At least when you consider a single node with a single point of failure; this is less true for a distributed system where you have redundancy.
At some point, you have to assume that either init is perfect, or that the code in the kernel to detect init failures is perfect, or that the watchdog monitoring the kernel is perfect, or whatever other layering you choose to put in place is perfect.
In a system with a finite number of components, there is always going to be a point at which you just say "this bit is going to have to be correct, and there's no other way around it".
You haven't gotten rid of a correctness assumption, you've just shuffled it around a bit.
We can't solve halting problem or verify program correctness for all programs. For a large subset of all programs, you are able to do both. This is a very important distinction.
We even have verified C compilers (CompCert). Writing a verified service manager should be easier in comparison.
If you send it a SIGTERM, it runs your shutdown scripts and reboots. If you send it a SIGKILL, your kernel will panic. As far as I remember, this isn't any different from init.
The OOM killer will _never_ take PID 1.
In runit all PID 1 has to do is run the service scanner. All that has to do is open directories and fork/exec the individual service managers. If it fails, it will try again in 1 second, forever. No complex logic needed. Just keep trying. In practice, it works surprisingly well.
If the individual service managers fail to run the startup script, it will try again in 1 second, forever. It works very will for most situations, but you _can_ customize this behavior easily. This simplicity is really helpful in an actual emergency because you don't get emergent behavior, like init deciding that your service is flapping and holding it down for 5 minutes.
Anecdotally, I've been using runit exclusively on all my systems (around 25 physical systems and 20-100 virtual ones depending) for at least 8 years now and I've never had a single issue.
The biggest problem I have with the design is that it puts your log services in a second level "behind" your main services, so you can sometimes miss that your log service failed to startup for some reason. This can be a real pain if your service uses blocking IO for it's stdout/stderr logging as it can cause the service to hang seemingly without explanation.
This was a few years ago and I don't remember all the details. We just had occasional issues with stopping/starting and especially restarting processes when pushing a new version out or when a process crashed. I do remember it could occasionally report a successful restart and still leave the old process(es) running.
All my developers became very familiar with supervisord, and it was number 2 on the troubleshooting list (1. Did we introduce a bug in a recent commit? 2. Did supervisord do something weird again).
After we switched to runit, only devs that touched ops knew about it at all. And we all forgot it was there. That's what you want in an ops tool.
(I know, as long as I'm using Linux, I'm stuck with dynamic memory allocation in the fucking kernel. I don't have a plan for how to fix that yet.)
(Also, thank you so much for all your help with Guix this last week!)
"holy crap I've got to recompile my kernel to get more processes" is so 1995...
It's cron and at implemented in a quite elegant way.
Unfortunately there does not seem to be much love for it in distributions.
Runit has the advantage that it is packaged in Debian and you can start using it right away.
The superficial differences are things like: service to logger pipe holding happens at different levels of the supervision tree (the root `s6-svscan' holds them in s6, the per-service runsv holds it in runit), `s6-svc -CMD behavior cannot currently be overridden whereas you can with `sv CMD', `s6-svscan' will immediately re-scan its directory with SIGALRM whereas `runsvdir' only polls for changes on a 5 second timer.
For basic supervision tasksboth are great, with runit being the simpler of the two in terms of understanding what it gives out of the box. For larger tasks (full system supervision, inter-service ordering dependencies, etc) s6 has the tools to make that easy whereas with runit you're going to find yourself playing stupid tricks in run scripts to get similar behavior.
monit checks for things, one of which is 'is the daemon running', and can then do things about it.
runit to keep things that crash occasionally running and monit to keep an eye on general system health would seem totally sensical to me.
(see also s6 which is another runit-like)
The things all daemontools-inspired process supervisors do: Runs a scanner gainst a service directory containing a directory for each program you want supervised (classically these are symlinks to another part of the system). The scanner spawns a supervisor program for every directory it finds which then looks for a program called `run' inside that directory and runs it as a foreground child. If the supervisor's child stops, it runs an optional script in it's supervision directory called `finish', and then runs run again. If one of the scanner's child supervisors stops, the scanner spawns it again.
In runit's case, the scanner is called `runsvdir', the supervisors are called `runsv', and it also comes with a program called `runit' that can act as a replacement for your PID1 init whose sole job outside of boot and shutdown time is to resurrect runsvdir if it exits. Note that runsvdir is perfectly happy to be run via an entry in your /etc/inittab if you're running under sysvinit.
At their most simplistic, that's all process supervision is - a bomb-proof way of keeping services running. All supervisors in the daemontools family come with a control program to interact with the process supervisor, as well as a stdin-based logger that doesn't rely on syslog. The main benefits it brings over the classic init.d/ model are simplicity (the most complex run script I have is 14 lines of dead simple shell, most are 4-5) and automatic restartability which init.d/ daemonization doesn't have.
From what I understand from monit's manpage, it's a full-bore rule-basd system monitor. Generally speaking, process supervision is tacking one problem (keep daemons running), whereas monit is tacking another (system state monitoring). Yes monit can act as a process supervisor, but it does so by polling the system state and hooking into the existing daemonization infrastructure.
I've been meaning to do some blog posts about runit, as it seems like it sits on the back-burner in general. Does anyone know how actively it's maintained, or has it just reached such stability that it doesn't need much maintenance? It would neat to see it on github with some real docs and such.
For containers it hits a real sweet spot: lightweight and easy to use within a limited scope of processes.
The vagaries of the OS wars make something like Runit tempting.
However, if you want to preserve the extreme flexibility of the daemontools approach, but with a workflow that is systemd-like (even converting systemd unit files to native service bundles), check out the nosh project: http://homepage.ntlworld.com/jonathan.deboynepollard/Softwar...
On the other hand, if you can ensure that your production systems all run relatively recent Linux distributions, you can safely assume systemd. Which specific distributions you have in production will determine the oldest version of systemd you have to support; since new features get added regularly, you'll want to know the oldest version you can assume.
What are you trying to provision, and what requirements do you have?
We can generally pin our entire fleet against whatever we want -- until we can't. One particular (very large, very important) dependency currently requires CentOS 6.5. Go figure.
As much as possible, I want to treat infrastructure as a commodity, so catering to specific versions (even LTR versions) breaks that goal. As another reply notes, if we lock in at Ubuntu 12.04 and it's predictable, that's excellent.
As another mentions, Upstart was easy to get a handle of quickly. As an aside, it bugs me that the (otherwise excellent and detailed) Upstart docs don't seem to mention that all of Upstart is now deprecated. Hence the original confusion! In general I don't want flexibility (not first and foremost), I want simple and consistent.
What dependency is that? And can you run it in a single-purpose virtual machine, rather than directly on real hardware, to make it easier to manage?
> As much as possible, I want to treat infrastructure as a commodity, so catering to specific versions (even LTR versions) breaks that goal. As another reply notes, if we lock in at Ubuntu 12.04 and it's predictable, that's excellent.
You definitely don't want to cater to specific versions any more than you have to; ideally you want as few versions across your entire fleet as you can. If you could get it down to just CentOS and the latest version of some up-to-date distribution, that would help; if you could make CentOS a virtual machine under that same up-to-date distribution, that's even better, insofar as you can then manage all the physical machines identically.
> As another mentions, Upstart was easy to get a handle of quickly. As an aside, it bugs me that the (otherwise excellent and detailed) Upstart docs don't seem to mention that all of Upstart is now deprecated. Hence the original confusion!
The Upstart docs aren't going to say that Upstart is deprecated. It's more a view of the Linux distribution ecosystem as a whole: the one distribution that primarily drove upstart usage and adoption is switching to systemd, so there won't be further momentum or resources behind upstart in the future. That doesn't make it instantly obsolete, but it does mean that starting a new project today around upstart is a bad idea.
(In fairness, there's one other notable distribution using upstart as well, namely Chrome OS; however, that's a bit of a special case in several ways, and in any case I hope to change that in the future.)
The Upstart thing is just somewhat odd, because it's a Canonical project, and Ubuntu is also Canonical's. At any rate, you're right that it's not immediately lost. But trying to pin against even the long-term releases of Ubuntu seems hopeless. I've been working on upgrading a 12.04 deployment to 14.04. There aren't any features that we're missing in the OS, so it's really only a concern from a security or support perspective. The shop I'm in was already using Ubuntu when I came on, but maybe the lesson here is to migrate to Debian, rather than bother upgrading to the newer Ubuntu release. (Migrating to CentOS would be somewhat more work.)
That's why they pushed so hard for Debian to use Upstart. Once Debian switched to systemd instead, Canonical announced that they would too.
> The shop I'm in was already using Ubuntu when I came on, but maybe the lesson here is to migrate to Debian, rather than bother upgrading to the newer Ubuntu release.
I'd certainly recommend that myself. Unless you have a hard requirement for an "Enterprise" distribution (RHEL or SLES), I tend to advocate Debian stable rather than Ubuntu LTS, especially on a server.
We also had a monitoring service (nagios) that would check if any services "auto-downed" on any servers and alert us.
#!/bin/sh
sleep 30
Instead, the way it was presented in public is that the Linux distros had been battling with brittle sysvinit scripts (true, but also largely self-inflicted) for so long until systemd came in to heroically save the day.
It was pretty shocking to watch how the scene evolved from apathy to having an urgent problem that must be solved now.
Upstart didn't come about until later from many of the alternatives I listed, and its origins were mostly in response to launchd. It was quite rudimentary initially. [1]
My first encounter with the existence of systemd was while keeping half an eye on the whole consolekit+polkit+udisk rigamarole i apparently needed to get thunar (or more correctly gvfs) to automount stuff.
This when i learned that consolekit was to be replaced with logind, and logind required systemd as init.
That, to me, was a very WTF moment. It basically made a file manager dependent on a specific init being used.