Immortal – A cross-platform, OS-agnostic supervisor for Unix-like platforms
immortal.run
immortal.run
And it presents a solved problem (some daemons fork themselves off and the parent exits, so tell them not to do that or use fghack) as a new thing that Immortal can solve.
Is there something actually new and interesting about this that I'm missing?
Mostly, though, covering the point about parent forks and exits that you mention.
The ability to have go produce cross platform binaries would ease a lot of deployment issues as well. Along with distributed logging, it's something I'll probably try using. Unless any of the above have advantages, primarily in the ease of use category.
The idea behind was to avoid a developer to deal with a DevOps/operator to just stop/restart services, something useful on staging environments.
For starting, stopping a process there is no need to unlink/link a file, this simplifies things a little bit, for example, https://immortal.run/post/ansible/ by only updating the file contents or "touching" the file it can be started or re-started.
When following a PID, signals can still be sent to the process, these things become handy while working with a lot of microservices.
The docs need to be improved, but any feedback, comments or even desired feature is more than welcome.
* There is no composable toolset, but rather one monolithic program that does only a small part of the job. envdir and setuidgid are -u and -e options to the supervise analogue that is that monolithic program, "immortal". All of functionality of the other daemontools tools, such as envuidgid and softlimit, let alone the likes of runit's chpst, is missing. Not only are there no composable tools for it, the monolithic program simply lacks their functionality entirely. One can extend this functionality with the composable toolsets, with tools to handle things from scheduling classes to control groups. The monolithic program lacks all of this.
* There is no equivalent of the service restart/finalization mechanisms of runit, perp, s6, or nosh; which permit taking various actions dependent from how a service program terminates.
* Nor is there an equivalent of the at-stop and at-start mechanisms of s6 or nosh.
* There is no service readiness mechanism, as s6 and systemd have.
* s6, nosh, and systemd all have the concept of "oneshot" services. immortal does not cater for them.
* The svscan analogue, named "immortaldir", lacks an equivalent to the "down" file mechanism from daemontools, runit, perp, s6, et al.. All services always come up at bootstrap. There is no enable/disable mechanism at all.
* The author hasn't borne in mind the design lessons from svscan, the reasons why it works as it does, such as only having symbolic links in the scan directory, not the actual service definitions themselves. There is a race condition between copying or saving the YAML file into its directory and immortaldir trying to open and read it, which can result in immortal reading a part-written configuration file. There is also the potential for hidden files to be recognized as services, something that svscan explicitly prohibits.
* The author hasn't borne in mind the design lessons from s6-svscan. immortaldir wakes up and polls every few seconds instead of being event-driven, which as M. Bercot notes is a consideration on systems that want to save power.
* immortal does not have logger services, as perp, s6, daemontools, daemontools-encore, freedt, nosh, and runit all do. Instead, it has only the options of the service manager itself writing the log file, or a non-supervised non-manageable logging command. It is thus not possible to control logging through service management.
* There is no equivalent to multilog's rotation scripts, filtering, multiple log directories, or automatic timestamping.
* The log rotation uses the old .1 .2 system. The author hasn't borne in mind the design lessons from multilog, the reasons why it did the things that it did. So it doesn't ensure that logs are safely written to disc at rotation time, for starters.
* The service management API is not compatible with daemontools, daemontools-encore, runit, and nosh; which all share a common binary-compatible subset (the daemontools service management API). This API is, rather, a WWW server. Where daemontools et al. send simple one-byte commands; this API uses JSON and the Internet Message format meaning that a simple command to send a signal is actually an entire HTTP/TCP transaction, with all of that parsing and marshalling going on, under the covers. This flies in the face of the "do not parse at runtime" dictum followed by the other toolsets. There is an entire HTTP parser and server running with superuser credentials inside the service manager here. Ironically, this is a criticism that people vehemently (but erroneously) level at systemd. This service manager actually does do this, though.
* It also contains an entire YAML parser.
* There is no service interdependency mechanism; at all. There aren't even the tools for the old "thundering herd" approach of starting all services and having them manually check their dependencies in their "run" scripts. There is nothing there akin to s6-rc or to the service bundle dependency mechanism in nosh.
* There's no mention of portability to the Hurd, Illumos, OpenIndiana, the Windows NT POSIX subsystem, the Windows NT Linux subsystem, and suchlike. Several of the other systems address the fact that "portable" to any "UNIX-like platform" means more than merely FreeBSD, macOS, OpenBSD, and Linux. s6 has notes on how to build it on Solaris, for example.
* The new and interesting thing is not the fact that it focusses on the solved problems of fghack and PID files, which ironically have been going away over the past decade. The new and interesting thing is spinning up service and supervise directories on demand from a YAML definition. nosh has the ability to spin up a service directory from a systemd service/socket unit file definition; but the other toolsets leave actually constructing the service and supervise directories from scratch to third parties. That said, such directories are simple enough to construct by hand directly, on the same order of simplicity as the YAML definitions here, in fact. It is notable that the YAML definitions again seem to have features weighted towards PID files and suchlike which are problems that are in the process of disappearing.
* One of the fundamental design concepts of the daemontools family is that the service manager knows the process ID of the daemon because it remembers it from when it spawned it and knows that it does not go away until wait(). immortal employs the old broken idea, that this replaced, instead. The code reads process IDs from PID files and tests whether they are valid by sending them signal number 0, in the race-prone and risky way that the world has known not to do for decades. Witness https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64... and https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64... .
There is a lot of prior art here that this design is just ignoring. Overall, it is a retrograde step, to a system that is less capable than runit or daemontools are, is less robust, and is less well designed. A better approach is, if one wants to drive services from a WWW browser and to create service definitions from a configuration file, to layer a WWW server and a service definition creation tool over something like s6, which is more complete and more widely portable than immortal.
I also recommend that the author read at minimum Daniel J. Bernstein's design doco for daemontools and Laurent Bercot's design doco for s6. M. Bercot goes into some depth when it comes to the reasons why things are engineered as they are. Additionally, the first chapter in the nosh Guide enumerates some of the design fundamentals that underpin it, which daemontools-family systems generally share.
I apologize in advance because unfortunately, the current documentation of the project don't cover all cases and because of that, some assumptions can be made until going deep into the code or testing, I will try to clarify some point hoping I could transmit a better idea of what the goal was and how is working, so pease bear with me.
For example, there is a "down" option, same as the "once" options https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...
Same with the enable/disable that in immortal is "start/stop" or also -x that is "exit"
You indeed can send a signal STOP, mainly because of this when using [immortalctl](https://immortal.run/post/immortalctl/) the option "stop" is used instead of down:
https://github.com/immortal/immortal/blob/master/cmd/immorta...
When following a pid, you indeed can still send signals to the "adopted PID" if we may call it that way.
Now immortal is totally capable of knowing the process ID of the daemon, but the methods you described as for reading PIDS from a file and checking if process are up and running, does methods are mainly used within the `-f PID` follow PID option in where you want to supervise or know status about a PID that the supervisor didn't create and, for this in BSD systems Kqueue is used instead of long polling https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...
I have some animations here http://immortal.run/about/ maybe they are not very clear but hope they could clarify this.
In case you know a better way of knowing if a process exists, something like asked here: http://stackoverflow.com/a/15210305/1135424, I will be more than happy to implement it.
Regarding the login options, I would say it was the main motivation to create `immortal`, http://serverfault.com/questions/752839/how-to-make-svlogd-s..., these days dealing with microservices and try to follow a 12-factor approach force you to innovate, in this case the idea of delegating a logger to a 3rht party is because doesn't limit your options, for example, if you want you could just pipe directly to [filebeat](https://www.elastic.co/products/beats/filebeat) and at the same time continue writing files to disk and rotate them.
The idea of not using link files was to simplify more the process, is true that while writing to the file there could be some issues, I will have to check/test more in detail, but the approach I took, was to keep checksum of the files to guarantee integrity and also based on the atime take actions: https://github.com/immortal/xtime, not perfect probably but is a start.
Regarding the portability, I think currently I am bound to the output and systems in where [xtime](https://github.com/immortal/xtime) could work, so basically, limits to, Darwin, FreeBSD, Linux, Dragonfly, NetBSD, OpenBSD,
Regarding the web server, the idea came by accident and I would say that thanks to the power of [go](https://golang.org/) this became very handy, mainly because daemontools/runit use named pipes for controlling the processes, but since they are not bi-directional you need two pipes, now the idea of using a Unix socket, is because they are bi-directional, therefore you just need one file and by "accident" you just inherit a nice capability that allows you to interact via HTTP with the socket.
http://immortal.run/post/nginx/
And well, for sure more work and improvements need to be done, including the bad documentation that may cause bad impressions about what really is wanted to be solved and cover.
One again thanks for your feedback and time on this, maybe we could use https://github.com/immortal/immortal/issues for special issues.
I strongly recommend the author of immortal to join the supervision mailing-list on skarnet.org, follow the discussions there and ask questions if needed. We have about 19 years of experience with process supervision, there has been a lot of prior art and successive refinements, and we hold extensive knowledge about what constitutes good practice. If someone aims to design a process supervisor, even if it's a completely new take with a specific spin, it would be a mistake not to tap into our pool of experience. We always want users to benefit from good, rigorous software.
For cases in where is required to monitor services like unicorn/gunicorn/Nginx that demonize them self and have the capability to fork how to make aware the supervisor of does changes and to continue monitoring the new process without entering into a race condition, to be more clear here is an example of this problem https://asciinema.org/a/80371, and work around's here http://serverfault.com/a/587065/94862.
Immortal and Systemd seems to fix this problem, but based on JdeB comments, they are employing the "old broken idea - Witness https://github.com/immortal/immortal/blob/9ed9087a8ef1e64d64...
Therefore wondering, what is the best approach, solutions for this cases?
Or could it be than Immortal is not that wrong, at the end, is solving a problem, maybe broken or not but is working and could be improved for sure:
As a reference, from systemd man page for options
Type= If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main daemon process. This is the behavior of traditional UNIX daemons. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can identify the main process of the daemon. systemd will proceed with starting follow-up units as soon as the parent process exits.
PIDFile= Takes an absolute file name pointing to the PID file of this daemon. Use of this option is recommended for services where Type= is set to forking. systemd will read the PID of the main process of the daemon after start-up of the service. systemd will not write to the file configured here, although it will remove the file after the service has shut down if it still exists.
The best thing to do is to contact the daemon's authors and insist they change that. In nginx's case, they think it's ok if they perform their own supervision, but it would be nice if they also could integrate with existing supervision systems.
In the meantime, as a workaround, the accepted practice for supervising forking daemons is known as "fghack": maintaining a process keeping track of the forking daemons via a few pipes that close when the daemons die. s6, for instance, provides such a tool: https://skarnet.org/software/s6/s6-fghack.html . AFAIK, so does nosh.
It's an aesthetic thing mainly, and a bit silly I'm sure, but dammit, I find it unsettling to have my Python 3 stuff running, and then the legacy Python 2, simply for the sake of supervisord.
Likewise, I don't need any Python on my Go-server, please.
I can see caring about performance. A Python process will be more memory hungry than something hand coded in C. But for the rest of it, who gives a crap if t is packaged well. And if it's not, it's probably not worth using it at all.
As I said, it's a question of aesthetics, and probably quite irrational, possibly OCD, but I do like my servers tight.
I don't have much experience with it on windows, but I use it regularly on *nix platforms and it works wonderfully.
That currently would be the primary barrier to usage -- the docs arent very helpful.
I want to use Immortal instead of systemctl. Just a question - is this mature enough to be used in production systems ?
the readme file has some instructions about how to do it: https://github.com/immortal/immortal/
Regarding if is mature enough, strictly talking still not a stable version, current beeing `0.11.0` and not jet a `1.x.x` but the more you could test and evaluate could be nice.
Please feel to raise any issue if needed here: https://github.com/immortal/immortal/issues
Is anyone ignorant enough to believe that any monitor can survive general process failure or contribute to recovery of a system with abnormal function or resource starvation?
And that is the basic reason for these monitors! garbage. My last experience was with bluepill which was this mess of ruby nonsense. Anyone with any experience knows you can't trust a system to correct it's own failure. Why are we hammering this nail over and over to catch a 30% recoverable transient...management.