Systemd has 6 service startup notification types, and they're all wrong | Hacker News Reader

Systemd has 6 service startup notification types, and they're all wrong

ewontfix.com

48 pointspmarin72 comments

Just add sd_notify, it's 65 lines of C code plus a couple of includes. I implemented in pure ruby with no problem either. See for yourself: http://cgit.freedesktop.org/systemd/systemd/tree/src/libsyst...

My biggest gripe is with socket activation. Systemd doesn't provide any info apart from the file descriptors to the application so binding to port 80 and 443, you have to know the order to add SSL on the right one.

this seems to be a common misunderstanding how apis and protocols work: you cannot just copy-past the internal implementation because that may change, the only thing that is stable is the public documented interface

in this case the public and documented interface is sd_notify and it is only available as part of a huge library that every service has to link to their binary

it would be nice if systemd authors werent completely ignorant about sensible interface design between software components

now that ppl started to depend on random internal behaviour of systemd it is no longer possible to reimplement it in a sensible way: nobody knows which systemd quirk your ruby project depends on

> this seems to be a common misunderstanding how apis and protocols work: you cannot just copy-past the internal implementation because that may change, the only thing that is stable is the public documented interface

It isn't an internal interface. http://www.freedesktop.org/wiki/Software/systemd/InterfacePo...

Some of the complaints about sd_notify is that it is just an implementation on top of $NOTIFY_SOCKET.... which is completely undocumented. So you have a decently documented and small implementation by Lennart that relies on something that might as well be hoodoo magic unless you've gone through the undocumented code.

There's also problems with nscd as pointed out in the comments, but the FSF is insane enough in general that I don't think that's necessarily a problem with systemd and more of an issue with working around the labyrinth of bullshit that is the FSF and their policies.

But $NOTIFY_SOCKET and how to use it is documented, see here: http://0pointer.de/public/systemd-man/sd_notify.html

???

That is documentation on sd_notify, which is an implementation that utilizes $NOTIFY_SOCKET. It being documented is not the same thing as $NOTIFY_SOCKET being documented

Option 1 is already provided. The sd_notify function is simply a convenience wrapper for sending messages on a socket specified by the NOTIFY_SOCKET environment variable.

See the notes in sd_notify(3) or https://lists.debian.org/debian-ctte/2013/12/msg00230.html for a simple alternative implementation. Admittedly could be better documented.

edit: point to notes section

MSG_NOSIGNAL is a portability PITA.

Well, pointing out what is wrong with systemd proves to be very easy for some people, but fixing the issues pointed out or even start building something different that doesn't have these flaws and still can innovate, is quite another story. There is definitely a lot of hate towards systemd (ignore the justified or not argument), but a complete lack of equally innovative alternatives.

Imho this is the semi-religious 'vim vs emacs' debate all over again, except this time it's not between equally good pieces of software but a 'systemd is garbage' group against the rest...

It's "systemd is garbage" vs "everything else is garbage".

> forking - assumes the original process invoked to start the daemon will exit once the daemon is successfully initialized, and not earlier. Requires a pid file and is subject to all the traditional flaws of pid files (but systemd can mitigate them somewhat by being the process that inherits orphans).

No, systemd uses cgroups to manage forked processes (with the option of pid files).

I'm not sure why I should seriously consider a critique of systemd by someone who doesn't understand how systemd works.

It uses cgroups to contain forking daemons somewhat and detect certain kinds of failure, but it doesn't use them to work out when the daemon's finished starting and is ready, which is what the blog post mainly seems to be about.

I think the fact that the article title has a capitalized "S" in "systemd" speaks for itself -- ie. the author doesn't know much about systemd nor has followed systemd development.

The other systemd mentions in the article are spelled correctly.

There are some capitalization styles that always start sentences with caps. It used to annoy me with "Coreboot" (instead of "coreboot"), too - but it really isn't a big deal.

I thought that a forking init script was supposed to wait for the service to be active before exiting. Otherwise you run "service apache2 start" and apache has not yet started after the command terminates.

This is the same requirement systemd has for the "forking" type daemon.

This, just because OP can't figure out how to forking doesn't somehow mean systemd screwed up.

Either the socket activation or notify approach don't need linking against a systemd library either they can just be done in < 20 lines of code.

Note the irony of the situation. Writing an init.d script is actually really hard. A post attacking Systemd is actually a pretty good argument for using it.

The common argument that systemd makes init jobs unportable sounds quite funny to me. I've used a lot of *NIX systems and init.d jobs are anything but portable, unless you fill them with OS-specific hacks.

The issue isn't that systemd makes init configurations unportable, it's that it makes programs unportable by tying them to systemd. What do we do 5 years down the line when we realise systemd was a bad idea and want to replace it with something else?

We have that something else, or maybe another component of the system implement the systemd notify api?

systemd has a vast number of APIs, especially inter-component ones, which are unstable and undocumented, meaning you'd have to replace the entire thing wholesale if you didn't want certain parts of it (especially its PID 0).

The thread we're on is talking about a public API that's not documented sufficiently for alternative implementations to be created without reading its source code.

Also you can, in principle, write a robust, automatic converter of systemd units into init scripts.

Based on my previous experience working on an init replacement, "forking" is probably not flexible enough. It turns out there are just enough daemons that either don't fork or fork but don't exit, but where you still need to wait for a pidfile, that it's a bad idea to assume one implies the other.

"wait for the service to be active" Well, how do you do that ? The service needs a way to to say "I'm done initializing and are now up and ready". That's not the way a lot of daemons are written today. They normally fork twice (or calls the daemon() function) and then starts initialization.

Looking through all the current init scripts, almost none of them do that. They launch the service executable, and report success it that succeeded. The service then often fork to place itself in the background, starts initialization, and might fail at that point without the init system noticing it.

The "forking" option of systemd will do the same, which is reasonable since a lot of existing services already works that way.

The apache init script on my system is one of the exceptions though, it doesn't directly launch apache, instead it starts apachectl, a program specifically written to start and interact with apache. That program communicates with apache, and therefore knows when it's started, whether it failed because httpd.conf had an error and so on.

That's not a general approach - unless you want to have each service accompanied by a program capable of doing IPC with the service for use with e.g. startup and shutdown. That'd make services portable among init systems and operating systems, so if you need that, it's a reasonable approach.

For systemd you'd can do the same, run apachectl to start apache.

"They're all wrong", followed by listing three mechanisms provided by systemd that get this right (socket activation, the dbus method, the notify method). The underlying problem is that there's no solution that's portable across init systems. That's not systemd's problem - ever other init system has also failed in this respect.

> The underlying problem is that there's no solution that's portable across init systems.

He describes one that would be portable. Even upstart's SIGSTOP hack is portable across any init system that chooses to implement it; no linking to any init-system-specific library required.

His option 1 is less featureful than the protocol provided by systemd, and is also not implemented by any other init daemon. His option 2, well. No. Doing it reliably means task-specific knowledge, which means adding the same kind of complexity that init scripts are currently full of. Maintaining this knowledge in the daemon is absolutely the correct thing to do. It would obviously be preferable to have one mechanism that works across all init daemons, but since there isn't one we're stuck with the current situation.

systemd's implementation is exactly as portable as upstart's hack - any OS can implement it. The relevant code is under a liberal license.

the original poster showed two ways to achieve a better result with less coupling. Given that systemd is already pushing a myriad of other mechanisms not implemented by any other init daemon, one has to wonder why you are even trying to dismiss the argument rather than engaging with it. Is it that you think strong coupling is good in systems?

No. Right now if anyone implemented option 1 in an init daemon, applications using it would be coupled to that init daemon. So it doesn't reduce coupling. It's also worse than sd_notify() from a technical perspective - there's no mechanism for passing metadata such as the PID to track, or information about the failure, or watchdog support and so on.

Option 2 is just shit.

I'd prefer a cross-init mechanism for this. But there isn't one. What should systemd do? Just not implement anything until SMF and launchd developers agree to add support?

Figure out the problem or create a standardised way of doing it rather than making whatever systemd chooses the standard, you know discuss things.

I think we can all agree that any mechanism is going to involve coupling; the question is, how much, and what limits does that impose?

The proposed 'use unix pipes' architecture proposed by the original author ('option 1') doesn't bind a daemon to a particular operating system or init system, because it uses a kernel feature common to all unix variants. And of course it itself is a mechanism for passing metadata, so your argument doesn't make sense.

Using a socket would also be fine, and if systemd chose to expose and document that socket API directly, rather than hiding it behind a C shim, that would be tolerable. But since systemd's modus operandi so far has been to attempt to become tightly coupled to every userland component, I'm not holding my breath.

The underlying NOTIFY_SOCKET implementation is defined as a supported interface (http://www.freedesktop.org/wiki/Software/systemd/InterfaceSt...), and is documented (http://www.freedesktop.org/software/systemd/man/sd_notify.ht..., see the Notes section).

That is not documentation, that is incomplete[1] notes in a man page of a library call. That said, that's getting pretty close to a correctly uncoupled implementation.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=820448#c3

The notes define that a socket name starting with @ is a Linux abstract namespace socket, so I don't see where "incomplete" comes from. You can quibble over whether the documentation meets your preferred style (and I'll admit it certainly is odd to reference the underlying implementation in the notes for the wrapper rather than the other way around), but it's a page of text, shipped with the code, that tells you everything you need to reimplement it.

The link I included in my comment shows that the behavior is not actually what is stated in the notes. Do you have any documentation that describes the complete behavior?

That's how Linux abstract sockets work. They're documented in unix(7).

oh man, that's a ridiculously grotesque hack. Huh! Thanks! You learn something new every day.

That said, now I don't understand why one would use these things instead of something less openly hacky, like normal portable sockets, which have all sorts of advantages. And I can't find a reason why these were implemented in such a weird and nonportable way. From all I can piece together from the web, it looks like these were implemented by the freedesktop guys for that hacky desktop bus thing, is that right?

Well that's an aside; in any case, having discovered this unnecessary coupling to a hacky linux-only feature, I would go back to the OP's idea of using standard unix IPC, which for the purpose appears to be more than enough to do the job but doesn't bring baggage. Any idea why that would be a bad thing?

Abstract sockets were added to Linux waaaaaaaaaaaay before dbus existed - they were already there in 2.4.0, which is as far back as the history I have goes. You really can't blame the Freedesktop people for them. If your init daemon doesn't run on Linux there's no reason to care about supporting them, either. Write a library that implements sd_notify(), and just leave that bit out.

Writing option 1 into a daemon is basically adding these SIX lines of code and a command-line option:

  #include <signal.h>

  ...

  int use_sigstop = 0;

  ...

    case 'S' :
      use_sigstop = 1;
      break;

  ...

  if (use_sigstop) (void)raise(SIGSTOP);

  ...

Option 1 explicitly discusses using pipes, which aren't hugely easier than sockets. The downsides of overloading sigstop have been fairly widely discussed.

I was responding to, "Rather than requiring library code to notify systemd that the daemon is ready, use some existing trivial method." It's even simpler and as a bonus the SIGSTOP approach is compatible with upstart.

But for a pipe it does get more complicated. Do you have sscanf? How about strtol? Should you use atoi? Might want to just roll a for loop of your own for three digit integers say. Or what if the other end of the pipe was closed? I have to ignore SIGPIPE. Should I use sigaction? Well signal should always be there. But is my daemon multithreaded? Shoot, looks like I have to fork and immediately exec a helper that does the write.

See where this is going? And the code someone linked to here for sd_notify style socket communication instead of linking with systemd (you are using pkgtools right?) used MSG_NOSIGNAL precisely for these signal issues. Guess what, it was different in Linux before 2.2 or something, can't remember the details. Also it's different in some FreeBSD versions as well. I'm pretty sure it was different on Darwin. I can't remember about Solaris. HPUX? AIX? Your guess is as good as mine. It's a portability PITA.

Anyway, I really really dig the simplicity of raise(SIGSTOP). Also yes there are some detriments to overloading the use of SIGSTOP like this, maybe a small performance one, and maybe there is some concern another root process could use kill to send it SIGSTOP. But I think the simplicity and reliability outweighs all that stuff I mentioned above if you go for a robust portable alternative using a pipe or socket. Plus I can make arguments in a similar vein about the environment variable that systemd uses internally which is not pretended with something like "SYSTEMD_" or "_" even, but I wouldn't cause it's just as silly as complaining about the detriments of using a signal in my mind.

Overloading SIGSTOP means the meaning of SIGSTOP changes, yet you say it will be reliable? Not very predictable behaviour. Also probably not UNIX like ;)

> However, none of the above choices actually make it possible to do this with a daemon that was not written specifically to interact with systemd!

Et voila, you found your answer. Target systemd, problem solved.

... and give up the ability to easily run on any non-systemd system, like BSD, Solaris, OSX, older/embedded Linux distros, and basically any other Unix. Oh, and it's GPL'ed so most of those systems won't use it.

Requiring systemd is like saying every daemon program needs a GUI.

You can't put systemd code behind a compile flag?

Who said anything about requiring systemd? All you have to do is handle it when it's there. It's no different than any other portability concern that has ever existed on any platform ever.

Rob had an excellent article about systemd http://www.landley.net/notes.html#23-04-2014

Is there a way, or any way to somehow stop systemd which is from Redhat? I'm very worried as I don't want to be _forced_ to use it on my embedded projects. Redhat does not know as much embedded system as their big gun server systems.

If you have issues with systemd in an embedded project, raise it so that systemd can be improved. Be a bit open in your thoughts. Note that systemd is used on various embedded systems already.

As stated in Rob's article, which I agree 100%, there is no point to 'improve' it, as it's something on the wrong track for me. IMHO, it's a disaster.

Not joking but we're moving our Linux-based stuff to FreeBSD (mainly operations stuff, mail, dev systems, cache and monitoring). We've done a lot of recent work and FreeBSD scales down better to smaller systems. Systemd isn't the only reason but it's a big motivator as we're really not interested in factionism and politics, just reliability and longevity.

Here we go again...

Dear pmarin, any specific reason why you bring up this critique from February now? Did something dramatic happen that makes this rant relevant again? Or are you just trolling? ;-)

To the OP: why don't you do something about it?

Judging by the comments to this post (and the links that the comments refer to), the author hasn't got all their facts straight. Also it seems like the author has some kind of personal hatred towards systemd, which doesn't seem to be based on real technical issues and has been writing similar vitriolic blog posts in the past.

This anti-systemd bashing is a complete waste of time. Don't like systemd? Don't use it. Do you think there is something wrong with it? Go fix it. Do you think it's done all wrong from the bottom up? Write a better replacement. But please, stop writing crap like this.

A lot of the comments are actually wrong though. For example, someone accused the author of lying for saying "There is no documented, stable way for a daemon to use either of these options without linking to D-Bus's library and/or systemd's library" - and as evidence, pointed to systemd's "stable interface" for startup notification, which is a C API that requires daemons to link against a systemd-provided library. Their own source confirms the blog poster's claims, and yet they accuse him of lying and having no scruples whatsoever for making them.

Edit: Interestingly, it looks like one of the other Wiki pages[1] does specificially call out the underlying $NOTIFY_SOCKET protocol as a stable interface. (Then points to an authoritive chart that doesn't, presumably because the implementation details of the other sd-daemon.h APIs aren't stable despite being documented in the same way?)

[1] http://www.freedesktop.org/wiki/Software/systemd/InterfaceSt...

Include this. http://cgit.freedesktop.org/systemd/systemd/tree/src/libsyst...

  ...
  systemd is free software; you can redistribute it and/or modify it
  under the terms of the GNU Lesser General Public License...

What if I can't?

The code for that is stable, it is just that the recommended way is using that library. When people asked, the code was provided to do this without linking. Just that linking is way more reliable.

The comment should've stated something like "the way to do this is not documented". The way it is currently worded implies something totally different.

> Just that linking is way more reliable.

Why? Is not linking somehow unreliable? You can't have it both ways. Either it is stable, or it is not.

Don't put words into my mouth to win an argument.

I'm saying that linking to a function is more reliable than copy/pasting that stuff into your program. If you want to argue that copy/pasting is totally fine, go ahead. I'm sure enough people will respond to that :-P

I believe the hatred for systemd has a few sources.

1) The first round was. This guy wrote pulseaudio!

2) This goes against the Unix philosophy

3) Systemd is taking over everything. Why replace init?

2) This goes against the Unix philosophy

Just like ZFS, Btrfs and SANs and cloud-systems in general. And they are really suffering in popularity for these terrible violations, eh?

After having fooled around with it, my only complaint as far as systemd goes is software needlessly making it a dependency. systemd itself is just fine.

I remember ZFS being lambasted for its "layering violations" a lot. Also SMF for its binary formats.

Now there's btrfs with the same type of "layering violations" and systemd ('s ecosystem) using binary formats.

What's good for the goose, ...

1) The first round was. This guy wrote pulse audio!

No argument there.

2) This goes against the Unix philosophy

Not a valid argument, nor true.

3) Systemd is taking over everything. Why replace init?

We all know what the problems with traditional init systems are, and they have to be replaced. Systemd currently appears to be the favourite option.

>This anti-systemd bashing is a complete waste of time. Don't like systemd? Don't use it.

This is increasingly becoming a completely unviable option. As more and more distros adopt it as the standard, and more and more software becomes dependent on it and it's various parts, it becomes increasingly difficult to just not use systemd. Particularly those of us in the enterprise world, where we have a limited number of distros that are acceptable for server related work, most of which are now set on systemd being their init system moving forward.

>Do you think there is something wrong with it? Go fix it.

Not everyone that uses Linux is a developer. Not every developer that uses Linux works in the realm of knowledge related to init systems. The group managing systemd is also known to be more than a bit vitriolic and opinionated - what you believe is wrong might be something they thing is right. This isn't necessarily a bad thing - but it does mean that the idea of just fixing it is impractical, even if you have the necessary skills to correct what you think is broken.

>Do you think it's done all wrong from the bottom up? Write a better replacement.

A fair amount of the people that think it's wrong from the bottom up think there are already perfectly acceptable replacements.

>But please, stop writing crap like this.

I haven't gone over the article in depth, but... why? Why are you so upset that people are using the avenue they have available to make their opinion known? The Linux community has far outgrown the days where it was a developer only playground, so your proposed ideas of just write some code to solve everything is ridiculous.

Personally, I'm past caring on the systemd issue - I've long since replaced all of my servers with BSD or Illumos based distros, so I don't have a horse in this race anymore. I just don't understand the visceral reaction of some people in the Linux community to go "STOP SHARING YOUR OPINIONS, IF YOU THINK SOMETHING IS WRONG JUST MAKE IT BETTER"

  > Why are you so upset that people are using the avenue they have available 
  > to make their opinion known? The Linux community has far outgrown the days
  > where it was a developer only playground, so your proposed ideas of just 
  > write some code to solve everything is ridiculous.

The author has the ability to write code. So his suggestion is relevant. But instead of writing code, he's writing highly opinionated "articles" full of claims that distort what is really going on. Opinions stated as facts, suggesting things by the way things are written, etc.

That'll just get you nowhere. Constructive but harsh articles are good. This? No

This is increasingly becoming a completely unviable option. As more and more distros adopt it as the standard

Distros has always had standards and this has never been an issue before.

more and more software becomes dependent on it and it's various parts

This is something completely different and unrelated from what distros decide to do.

For instance, while I really like systemd as a Linux-specific init-system, I cannot for the love of all that is good architecture understand why Gnome, a desktop-environment running on top of X, maybe running on top of Linux, should make it self dependent on a Linux-specific component like systemd.

In cases like that you should not hate systemd, but on the guilty party introducing the bad dependency. This is clearly not systemd's fault.

>Distros has always had standards and this has never been an issue before.

While distros adopting standards has been polarizing in the past, I think this is the most debated issue in the past decade or two. There's been some quibbles over things, and of course there's the vi vs emacs holy wars, etc, but I can't think of a single time where something so integral to the distribution has been such a matter of controversy across the community.

>In cases like that you should not hate systemd, but on the guilty party introducing the bad dependency. This is clearly not systemd's fault.

Oh, I'm not blaming systemd for this (Though it's not surprising Gnome has done this - the Freedesktop.org group is fairly incestuous when it comes to this sort of thing), I'm just saying it's pretty easy to say "Don't use systemd if you don't like it then!", but increasingly more difficult to actually do it.

It's not like vi vs emacs or nginx vs apache where you can quite easily just use the one you want.

> understand why Gnome, a desktop-environment running on top of X, maybe running on top of Linux, should make it self dependent on a Linux-specific component like systemd

GNOME doesn't depend on systemd. See for instance a blogpost about it: http://blogs.gnome.org/ovitters/2013/09/25/gnome-and-loginds.... GNOME wants either logind (systemd) or ConsoleKit (freedesktop.org, unmaintained).

Calling people "guilty" is also not very constructive behaviour, IMO. Too emotional while you're already showing lack of knowledge about this subject.

"If you don't like it, write something yourself" is a perfectly legitimate answer, and people who have a problem with that answer are basically suggesting that we should just stop everything we are doing and write something just for them, which makes them entitled princesses.

> why don't you do something about it?

You've picked the wrong guy here. Rich Felker has done extremely valuable and useful work[1] for the Linux ecosystem. You know, unlike Lennart Poettering who brought us pain through PulseAudio and systemd.

> the author hasn't got all their facts straight [...]

> Also it seems like the author has some kind of personal hatred towards systemd, which doesn't seem to be based on real technical issues and has been writing similar vitriolic blog posts in the past.

His posts are purely technical and of extremely high quality, and if you can't see that, then this forum has failed.

[1] http://www.musl-libc.org/

Fully agree here

> His posts are purely technical and of extremely high quality

Don't understand how you can claim this. The posts are highly emotional, very suggestive and a lot of things are plain wrong.

> if you can't see that, then this forum has failed.

Whatever. That's not an argument.

Reply on news.ycombinator.com