Daemonization in Unix programs is probably about restarting programs | Hacker News Reader

Daemonization in Unix programs is probably about restarting programs

utcc.utoronto.ca

86 pointsingve59 comments

Open articleView on HN

> various bad things happening to it when, for example, you logged out (which often would deliver a SIGHUP to it)

And thus, `nohup`.

Quite a few of my self hosted things are spawned using `nohup ./foobar &` and happily run for months at a time.

Edit: TIL `daemonize`, I have a new go to!

https://software.clapper.org/daemonize/

I just bang out a quick systemd service and let it do all the lifting

Would you give a short explanation of how you would try that?

You configure it as a service,

https://www.freedesktop.org/software/systemd/man/latest/syst...

A random tutorial,

https://medium.com/@benmorel/creating-a-linux-service-with-s...

Thanks. I’ll look it over

I'll do a system d but a problem is (1) system environment, and (2) machine specs.

To normalize machine specs I will run it in a VM on the cloud.

To normalize system environment I run in docker.

I also like to run on Java for code safety and consistency.

So what I have is a VM (Java) running on a VM (Docker) running a VM, running a VM (cloud compute), running on system D. I don't think it's abstracted enough though so I'm looking to find ways to add more VMs to the equation, so as to ensure consistency. Figure if I add a few more VMs, good luck hacking that!

Worth to mention runit - bit bigger thing but still smaller than systemd

I love that Siebenmann is taking the time to document the nitty-gritty details of the evolution of Unix. Systems like 4.2BSD are long gone from the market, but we still see their influence all over the place.

"these days it's somewhat controversial and sometimes considered undesirable"

By whom and for what reason? Every non-trivial OS has some form of "daemon" concept, regardless of what name it's given. What alternative is proposed? All I get from this statement is discussion about the deficiencies of how demonization is performed.

[edit] I get it. Don't deamonize yourself. That's fine, and a good idea: it's frequently done wrong and even when it's done right things change and otherwise correct software is suddenly wrong again.

They just mean service managers like SystemD and OpenRC prefer to handle daemonizing themselves, and thus would prefer that your program stay in the foreground and let them put it in the background.

From OpenRC’s docs[0]:

  Daemons must not fork. Any daemon that you would like to have monitored by supervise-daemon must not fork. Instead, it must stay in the foreground. If the daemon forks, the supervisor will be unable to monitor it.

  If the daemon can be configured to not fork, this should be done in the daemon's configuration file, or by adding a command line option that instructs it not to fork to the command_args_foreground variable shown below.

The “undesired” or “controversial” part is whether programs should do it themselves or not.

[0]: https://github.com/OpenRC/openrc/blob/master/supervise-daemo...

deamontools was the first supervisor service I used that required that programs not background themselves, it even included a tool for preventing "legacy" daemons from doing so.

It made a lot of sense to me at the time and honestly felt easier. Going back to init.d or upstart just felt like a step backward and so much more complicated that it needed to be. Then SystemdD comes along an have the same expectation and things makes sense again and writing "startup scripts became as easy almost as it was with daemontools.

inetd, the "super server" from 4.3BSD, supported this in addition to handling the listening socket. For reasons I don't fully understand, inetd fell out of favor despite having been installed and running by default on pretty much every *BSD and Linux server for decades.

I recall that inetd started a new instance of the demon for every incoming connection, and this caused lots of processes when lots of connections happened.

I don’t recall whether you could tell inetd not to do that.

inetd could pass the listening socket to the process. That was the `wait|nowait` field in /etc/inetd.conf. The typical config for TCP used with services like finger was `nowait`, which meant inetd would listen on a socket and spawn a new process for every incoming connection, without waiting for a previously spawned process to exit. But in `wait` mode it would spawn the process when it detected a connection, pass the listening socket (not connected socket) as fd 0, then wait for the server to exit before polling the listening socket again.

inetd was (remains?) a perfectly useful solution in this space. It just maybe needs some love to add some convenience features. Off the top of my head: 1) ability to split /etc/inetd.conf into, e.g., /etc/inetd.conf.d; 2) ability to trigger a restart of a specific service, rather than restarting the entirety of inetd.

Couple things come to mind.

One, is that HTTP took over the role of a lot of simple servers, thus something like Apache and CGI-BIN was used in place of inetd.

Second, with the rise of interpreted languages (i.e. Perl et al), forking became more expensive. With binary programs, forking tends to be cheap (in a multi-use case) since the executable page are shared across processes. While the interpreter runtime itself is shared (being compiled), the actual script is not (having to be loaded for each instance).

The HTTP servers managed that better (through modules, FastCGI, etc.), so that space didn't really advance under the guise of inetd.

Make no mistake, an inetd service is "fast enough" for a wide array of use cases today, both compiled and interpreted, simply because the hardware is much better today. But, still, when folks think "ad hoc" service today, they're likely turning to HTTP today anyway.

Daemontools is tiny, systemd is huge, but the only thing I like about systemd, daemontools could also do.

Have you seen s6?

I’d say it’s the modern-day equivalent of daemontools, and it’s under active development.

https://skarnet.org/software/

> By whom and for what reason?

By everyone and for the reason that service managers lose track of the process, so it increases program complexity for a net negative in usability.

> Every non-trivial OS has some form of "daemon" concept, regardless of what name it's given.

And none of that is relevant, TFA is about the unix self-daemonization pattern, that is what's undesirable.

If your service manager cannot track forked process, that would be quite an issue anyway

"Daemonization" is nothing but fork + exec (and exit from the parent)

From the service manager, there should be few distinctions between this and, say, pgsql forking to handle connections. In both cases, I expect all child processes to be tracked.

And please do not tell me the "health check" part : having the main process alive is a broken health check, just like having a server powered-on is a far from enough

There is one very important distinction.

If pgsql forks to handle connections, the PPID of the child processes points back to the "main" pgsql process, which is still around.

Whereas, traditionally, if a wannabe daemon forks and exits, the PPID of its now orphaned child reverts to 1.

afaik you need cgroups if you want to accurately track a process tree regardless of what silly tricks the processes might be try to escape

Or prctl and PR_SET_CHILD_SUBREAPER are potentially useful for that.

So, all-round agreement that you need Linux extensions to fix legacy Unix situations that can be avoided by not forking grandchildren and using pidfiles and whatnot.

Yes, as time goes on, new features are added, and some are meant to address deficiencies that were unanticipated at the time and the limitations of the earlier designs are discovered.

How does that help?

Setting that on the init process is pointless, since init already has the reaping behavior, and it when a process is reparented it doesn't give init any information about what the parent was when it is reparented.

Setting it on the service process isn't helpful either, since the fored process will get reparented to init after the forking process dies since a dead process can't reap it.

With subreaping as the only additional mechanism available, init would have to fork itself first and then keep that child running until all its descendants are done. Basically, every service would have a supervisor process (controlled by init) which roots a tree of all its other processes.

Setting a process as the reaper means descendent processes are reparented to the reaper, which has to call wait like pid 1 does in order to collect the process's info and remove the zombie. This changes the default behavior where they are reparented to pid 1.

You don't set this on pid 1 because, as you point out, pid 1 is already the reaper of last resort.

You don't set this on the service process either; it is set on a process manager that isn't pid 1, so that process can monitor and collect info about its subtree of processes.

And not even cgroups are enough if the process has write access to another cgroup (for example, because it is root)

From the service manager point of view, a daemonizing program is a headache.

The program that the service manager thinks is starting immediately quits, and it's the grandchild (or even great grandchild) that's actually the payload service.

Because the parent quit, the grandchild becomes reparented to the init daemon; it is not a child of the service manager.

The service manager has no idea what the process ID of the service is.

If the self-daemonizing program writes its PID to a pidfile, and the service manaager is configured to know what that is and look for it, it can be obtained that way.

Not being the parent of the process, the service manager doesn't get notified when that process terminates. It has to use hacks to poll for it; like kill the PID with signal 0 to check liveness, and pray that the PID isn't recycled.

If you're writing a deamon program, make sure daemonization is optional, favoring opt-in over opt-out.

cgroup is the only way to track a process and its children, as far as I know

Either you use cgroup (in which case, neither the parent pid nor pidfile nor even the process ID are useful), either you do not (in which case, you leak processes)

Stopping a service means, in the end, killing all processes in the associated cgroup.

Deamonizing means double forking processes to break away from the terminal executing it.

This sort of behavior makes writing proper systemd unit files and containers a pain. It is a lot nicer to keep them 'attached' so that you can do things like capture stdout for logging and all that fun stuff.

That is what they are talking about when "daemonizing is considered undesirable".

I would be totally fine if Unix programs never detached themselves from their invoking terminal/process and that restart was handled by an external system. Honestly, that seems much more consistent with the unix philosophy than having a billion different services all with their own daemonization code.

This belief is mainly because after many years of debugging services, the very first thing I do is run the service outside of the daemon manager, with debugging enabled, so I can strace the process from its beginning.

`inetd` was an early attempt at avoiding every process having to write their own daemonization code.

not really. inetd was an early attempt at avoiding every process dealing with the basics of waiting on a socket for a connection, forking off some process to deal with it and managing the socket and the connection.

inetd waited on "all" the sockets; when a connection came in, it started the configured application to deal with it.

daemonization didn't really figure into it, AFAIR.

I thought inetd let you conserve memory because rarely-used daemons don't need to run continually.

This is the process advocated in this wiki:

https://mywiki.wooledge.org/ProcessManagement

Seems reasonable to me. Restarting processes is just something like this:

  #!/bin/sh
  # service.sh

  while :; do
    /my/service >> /var/log/service 2>&1
  done

  service.sh &

The service is just a normal program. If run directly, the inputs and outputs are attached to the terminal. If not, they're redirected to log files. Programs shouldn't care much what their outputs are connected to. Maybe they should turn off terminal escape codes in the outputs if they're not terminals but that's about it.

This approach has many problems; in particular, it lacks an intelligent backoff. A looping service will get continuously restarted.

It doesn't handle service dependencies. What if I only want my service to run when I'm on network and VPN is disabled?

And I can think of a lot more. All these things have been implemented in init, or in systemd, for decades.

I agree with you. I use systemd myself and think it's great.

It was just an example of how simple things can be when programs don't daemonize. Supervising and managing the processes is systemd's job and it does it well. Daemonizing makes its job harder and is not necessary to begin with.

There is another reason for a service to daemonize itself: you can have the parent process wait for the child to be fully ready before exiting. Modern init systems often have a better way to communicate readiness (for example systemd-notify, or writing a pidfile when ready) but older systems often didn't have a way for a process to directory tell the init process it was ready.

systemd has kind of "killed" daemons and the need to write custom daemon code other than IPC stuff. Which is both good and bad. I really like that a lot of user space tools just create a secondary binary and an easy interface to communicate to that process using the main binary, you can either run it by itself, use it as a systemd/sysvinit service, or one-off it.

for example I use my WM to start `swww-daemon` and then running `swww <path/to/wallpaper>` just sends a message over a Unix socket to the daemon process which handles everything else. Its sooo much better than internally forking off and all of that, it becomes messy quick and its pretty unnecessary in the present. All that really matters is how robust your IPC and stop-start process is. Sometimes its massively fragile, and it shows.

side note, when I first started Unix I was fascinated with daemons. I still think they are one of the coolest things ever.

Is there an archive link for this by any chance? I got a 403 error trying to load it with this message: "You appear to be trying to break this web server. Goodbye."

Yep! You can find copies of the article both on Wayback Machine and on archive.ph https://archive.ph/

"these days it's somewhat controversial and sometimes considered undesirable"

I respectfully disagree.

It it fundamentally related to your use-cases, execution times, and design paradigms.

If your team comes from a *nix background, than having dozens of processes interfacing over various pipes/network-sockets/fifo/mem-share is normal. In a cluster environment, having the ability to spin up processes on any host, yet remain functional over middle-ware can be very robust from a maintenance perspective.

A periodic self-restart trigger feature is only a small part of these feature sets.

Containerization on the other hand was necessitated out of poor OS design consistency, and rotten permission handling. It added overhead, complexity, and costs. However, it also solved a very real user requirement of keeping many things operational at the same time.

Daemons are almost certainly still required if you have software that must remain running for >6 months outside a users session. Also, periodic processes do not necessarily have to remain resident in all use cases if that was a concern.

Best of luck, =3

practically every system is running service manager of some sort (supervisord, daemontools, runit, or infamously systemd etc), and with a service manager daemonization is something between completely useless and actively harmful. as cks explains in the post, daemonization is truly relevant only if you are starting your daemons directly somehow somewhere, and that went out of fashion somewhere in the turn of the century.

Programs that deamonize should have a way (e.g. command line option) to opt-out of it, or opt-in. Unconditional daemonizing interferes with debugging, for instance. That's bad.

But don't forget that the service managers themselves have to daemonize. :) The programs which they launch don't have to daemonize only because their parent did that already.

Interesting, I think we treat system services differently than utilities.

A given host assigned capabilities should not require manual intervention, but should allow manual interaction for testing and debugging.

Insisting one method is superior, is often from people that have never hit the terminal spawn limit on a host.

Best of luck, =3

> the newly started program could inherit all sorts of things from your login session. It might have some random current directory, it might have stray file descriptors that were inherited from your shell or login environment, its standard input, output, and error would be connected to your terminal, and it would have a controlling terminal, leaving it exposed to various bad things happening to it when, for example, you logged out (which often would deliver a SIGHUP to it).

These things still matter when the program is run from a boot script or init daemon, rather than a login session!

daemonization is the process of running a process in the background and without being attached to a terminal. Many applications exit when they are detached from the terminal they starter even when they yield no output. Maybe that is why it's still there. Of course back in the day fork() was all about creating background processes that were running quietly in the background were screen didn't exist yet and terminals were actual machines, not just one more window. Today System D doesn't like daemonization. IT prefers a process that can be attached somewhere.

> without being attached to a terminal

Isn't that more the job of `nohup`?

No, nohup, as the name says, is about filtering/ignoring signals, specifically HUP, that are sent when the parent process shell exits.

No. While nohup redirects standard input and output and block the SIGHUP signal, it does not actually detach the process from the TTY session.

And this is why God invented tmux.

I prefer non-daemon things that I can use systemd to manage with Unit files Type=Simple. Very easy, sends things to journal for logging. All very nice and clean. Self-daemonization is anti-pattern for new software. Non-UNIX DOTW.

i always thought it was "do the collection of things to mitigate the problems that have been learned over the years that cause problems with long running processes"

changing the cwd so it doesn't cause problems with mounting/unmounting comes to mind.

Daemons suck if you're not in the microservices camp.

Can you explain further? daemons are OS-level and predate microservices by like a half-century, so how are they relevant to microservices?

It's just yet another example of someone spouting off about microservices with near-zero understanding of microservices.

If you're administering a server, what else is there?

There are service managers like systemd and shepherd, which use daemonization under the hood. Then there's container orchestration like docker compose, kubernetes, etc. For my toy servers at home I just use screen and have it set up automatically at boot.

What do you use outside of the microservices camp?

We've had init and inittab for decades. Daemonization was never a necessity.

Service managers don't daemonize anything, because they never have to release control to their parent. Just like init always did.

The only thing that's changed with systemd is that inittab now has a dependency management, and richer syntax.

Reply on news.ycombinator.com