Hacker News Reader

I don't see a boycott happening. Not unless someone steps in to write an alternative init system, which solves some of the issues openrc and systemd solve, but without systemd's baggage.

taking the discussion out of the opinion level to a more factual level(eg. i like the boot speed of my archlinux machine but the whole consolekit(why did we get that to begin with ?)/logind/ridiculousness, logging and service files vs. startx and grep /var/log is annoying to say the least, but as i said subjective)

on a factual level we have ridiculous bugs specifically caused by systemd that take down your system, and the lead developers saying either i don't care or it's by design. let me bring a few examples:

https://bugs.freedesktop.org/show_bug.cgi?id=76935

> That is the expected current behaviour, "debug" can cause "too many" messages to be useful anymore if things are broken.

or this one, systemd segfaulting with cgroups off:

https://bugs.freedesktop.org/show_bug.cgi?id=74589

Lennart:

> To make this work we'd need a patch, as nobody of us tests this.

There were a few other recent ones. When your new init system consistently takes down your operating syste, because of stupid bugs, and the lead developers say we don't care, and even block efforts to fix these issues, you should indeed worry.

https://plus.google.com/111049168280159033135/posts/Kd57G8s1...

In fact Greg KH jumped in to play daddy, and wrote a patch where tom gunderson commented that that patch would not be merged as it's not clear what problem it solves.

edit: some say openrc is the one. here's a comparison:

https://wiki.gentoo.org/wiki/Talk:Comparison_of_init_systems

> solves some of the issues openrc and systemd solve, but without systemd's baggage.

I thought OpenRC was that init system? Or what systemd baggage comes with it?

OpenRC is better than sysvinit, but still isn't the best. However in terms of technology, systemd is the best init system available. The main problems are the developer attitudes and architectural decisions they've made (big blob, work with me or nothing).

Openrc does not provide a pid 1 init implementation. On most openrc systems sysvinit launces openrc to handle service startup.

runit (from daemontools) might not start things up in parallel, but it's a whole lot simpler and reliable. Phusion uses it instead of upstart in their base docker VM.

Also a lot of enterprise shops use daemontools and runit, because they just fucking scale.

I concur, runit absolutely rocks. Haven't tried systemd yet, I wonder if the two play nice together (if they don't, it won't be runits' fault, that's for sure).

Might try the old-school daemontools, even though the commands are a bit different, it might be less work to get something going.

From your other comments I can tell you're a seasoned daemontools user, so let me just put in a quick plug for daemontools-encore [1]. Like runit, daemontools-encore can now run services in separate process groups with setsid(1). Why is this so darn important? Well, under classic daemontools a ./run file with a pipeline in it -- or really anything that forked -- would spin off orphans every time you did `svc -t|-k`. Very bad stuff to deal with in a production environment.

[1] http://untroubled.org/daemontools-encore/

Yup, it's one of those annoying things that makes run scripts more complicated. I rewrote phusion's runit setuser helper in Go because it didn't set all the right env vars.

https://raw.githubusercontent.com/steakknife/my_init/master/...

Huh, looks pretty similar to envuidgid(8) from daemontools. [1]

Another plug for daemontools-encore (last one I promise). The setuidgid(8) program in classic daemontools doesn't set supplementary groups, and a lot of people I know just end up just using sudo in their run files. If you use daemontools-encore though, you can just use `setuidgid -s` instead.

PS. Really, daemontools-encore should imo be called daemontools 1.x, but you know how djb feels about forks of his projects...

[1] http://cr.yp.to/daemontools/envuidgid.html

Maintainers either will or won't accept that forks are the sincerest form of flattery.

The problem as per the Phusion blog article is that the runit and possibly daemontools equivalent commands don't set all env vars: HOME, USER, GID & UID, it only calls setgid() & setuid() IIRC. (envuidgid only sets UID and GID[0]) This causes breakage for lots of apps that end up inheriting root's env instead. :(

Here's the go version I wrote, so there's no need for dep on Python/Ruby/etc on the target system:

    setuser USERNAME COMMAND [args...]

    # how to build it
    go get   github.com/steakknife/my_init/setuser
    go build github.com/steakknife/my_init/setuser
    # creates setuser exe here

[0] FR issue submitted as https://github.com/bruceg/daemontools-encore/issues/18

> base docker VM

you serious? criticizing systemd by promoting a totally new and un-tested software written by a single (for-profit) company.

> Also a lot of enterprise shops use daemontools and runit

provably not true. most enterprise shops run RHEL/CentOS which, to keep support contracts, runs sysvinit or in RHEL/CentOS 6.x ran Upstart. RHEL/CentOS 7 uses systemd and is not going to look back. Don't forget, the "serious" distros have used sysvinit for a very long time. Then they tried out Canonical's Upstart for a bit and decided it was not a step forward, but rather a step to the side. Now they have landed on systemd for very good reasons.

No, sorry, but you don't know what you're talking about.

daemontools is battled tested. We used it at Stanford on thousands of boxes. I used at a lot of shops, one in SF for example had 40k boxes (CentOS) and 6 ops staff, deployed via CFEngine. Heroku uses daemontools. On and on. Daemontools is mostly used when there is no desire/time to replace the existing init completely.

runit also scales amazingly well, even though it's more work to deploy.

Upstart is too unpredictable (flapping) and sysvinit doesn't do supervising very well, which is why important services are deployed with daemontools/runit.

I'm from Phusion.

Runit is great as a tool to manage your own daemons. But as a general init system for distributions? Not so much. Just take a look at the /sbin/my_init system that we wrote for baseimage-docker and what kind of functionality it adds on top of Runit, to give you an idea of why Runit by itself is not enough. Runit also performs no dependency management (i.e. starting one service before another) so it's quite painful in certain situations. For example, if you have a background queue daemon that depends on PostgreSQL, then you have to manually make sure that the daemon is not started until PostgreSQL is available. Otherwise you get tons of useless error messages as your daemon keeps getting restarted.

No problem, just wrote a tool for that:

    waitport [-u] port [timeout (float)] # -u = UDP instead of TCP


    go get   github.com/steakknife/my_init/waitport
    go build github.com/steakknife/my_init/waitport
    # creates waitport bin here

Renamed to https://github.com/steakknife/devops_toolchain to avoid confusion.

A collection of individual tools that can be cherry-picked to solve nitty-gritty problems.

> on a factual level we have ridiculous bugs specifically caused by systemd that take down your system, and the lead developers saying either i don't care or it's by design.

Agreed. This is a big worry. When the lead developers of something this important sound this callous, I start getting concerned.

I know you didn't get the chance to read this, but it specifically says:

Disclaimer: We are not sysvinit purists by any means. We do recognize the need for a new init system in the 21st century, but systemd is not it.

And I don't think that's an invalid point. I totally agree that a better init system is desirable, but I've also personally really struggled to understand why systemd is a good approach – it does seem, from someone who's not that close to the issue, that it's an extremely complex and monolithic approach to the problem.

> Even if it's technically wrong: Unix as a platform is now Linux. Everything else is niche products [sic], and we don't have to use the same init mechanism on Linux/BSD/whatever just because we can.

This statement is so utter in faultiness, I don't even know where to begin. In the first statement, you say “Regardless of the truth of it; this axiom X is true.”, and introduce a logical paradox, similar to the sentence “This sentence is not true.”[1]. Consequently, you call Mac OS X, FreeBSD and alike niche. The former is the second most popular commercial desktop OS, and the latter is incorporated into the former and also is the OS that powers PlayStation 4[2][3].

You say that “init systems need not be compatible”, and I say yes. Also, OS interfaces need not be compatible. Implementations of C library need not be compatible either, nor implementations of C themselves. Why did the size of a byte were standardised anyway[4]?

> I have deployed the same software on Debian Linux, RHEL Linux and AIX and basically had to reimplement service management scripts for every platform to have the best possible result.

You wouldn't have to if there were a standard initialisation interface, would you?

> Quoting the "Unix philosophy" as the reason to keep things like they were 20 years ago is laughable. I want to get things done, and managing processes using init scripts and daemonization is pain.

You want to get things done today; I'd guess you'd also like to keep thing happening 10 years later, when Linux is no longer as popular (Software come and go, no-one can guarantee that, say, illumos[5] will not become more popular than Linux in 10 years). Would you rather run around porting your init scripts to the new OS a cloud provider fancies, which your boss fancies, or write new bugs (which some refer to as features)? Would you rather wait for the mainstream repos of your favourite projects to catch up, or just deploy the program and enhance the experience? Would you rather read pages of documentation for your new incompatible init system, or skim the frontpage of HN?

> We start to rely docker and all sorts of new approaches to manage our services and nobody complains.

Well, Docker is a containment and distribution solution and programs need not be modified nor need to have around Docker to run. It is reminiscent of a FreeBSD jail or a plain old VM.

> Apple introduced launchd and nobody complained.

Just compare the source trees, you'll be enlightened[6][7].

[1] http://plato.stanford.edu/entries/self-reference/

[2] http://www.phoronix.com/scan.php?page=news_item&px=MTM5NDI

[3] http://www.vgleaks.com/some-details-about-playstation-4-os-d...

[4] https://en.wikipedia.org/wiki/IEC_80000-13

[5] http://wiki.illumos.org/display/illumos/illumos+Home

[6] http://cgit.freedesktop.org/systemd/systemd/tree/

[7] http://www.opensource.apple.com/source/launchd/launchd-842.1...

edit: format references

Doesn't illumos use something similar to systemd, called SMF[1]?

[1]: http://www.illumos.org/man/5/smf

Well, I just picked a random Unix I've heard of, I don't know a lot about it.

> You wouldn't have to if there were a standard initialisation interface, would you?

My point is, even right now with sysvinit (and aix doesn't use it afair), i have to adapt a init script to every single unix/linux distribution i want to use my software on. At least, if i want to comply with that system's best practice.

With systemd and other init replacements, it's likely a few lines of config file i have to rewrite. Its much less effort to ship or adapt to a few of these than have a init script that works everywhere.

About other unixes: calm down, this is some process that supervises other processes, this is in no way comparable to making libc incompatible. I assume we're talking about servers here that run off the shelf distributions. For (virtual) appliances and embedded devices / firmware everyone will roll their own images with whatever init they're happy with. We don't have any influence on what commercial vendors do. If AIX feels like an 80s system (i know it's modern and powerful at the core) that's IBMs decision, but i don't hear any complaints that Linux distributions don't ship ksh in vi mode as /bin/sh anymore.

So, in this particular market and with the problem at hand, i don't see how and why Linux should wait for the sysvinit replacement that is able to make everyone happy and will magically be adopted by everyone.

And if some day illumos will succeed Linux, I'm sure people will manage to port their 10 lines of service config over.

> [And] if some day illumos will succeed Linux, I'm sure people will manage to port their 10 lines of service config over.

It is not ten lines of service config being hard, though, it is grokking a new, ad hoc tech, which is necessary to get the most out of it. I have not a rage against systemd in itself; it is too low level for me to care about, process management that is, yet I'd support, to the last drop of my blood, a standard to cover initialisation. If I have to write an init script/config/whatever one day, I'd rather use a standard format that I can get to work on any system, than an ad hoc, OS-specific technology that I have to re-learn every time I switch operating systems.

You seem to overlook the importance of a process hat supervises other processes. I reckon that it should be so stable, one shouldn't notice it. That is possible if and only if the thing is tested and implemented according to a standard throughout long times.

"Quoting the "Unix philosophy" as the reason to keep things like they were 20 years ago is laughable. I want to get things done..."

The Unix philosophy (loose coupling of specialized pieces) is what permits you to get things done. It is the foundational theory behind the most successful architecture on the planet.

systemd is doing things the Windows way -- monolithic, tightly coupled systems wherein, using Occam's Razor, the explanation for some of the coupling has to go into the realm of the nontechnical.

Those of us who have to manage and create large complex systems sort of scratch our heads at the idea that desktop systems booting faster is an important goal deserving of all this mess. Linux on the desktop has always been terrible and will always be terrible, and getting to it a few seconds faster is not helping you.

Windows has historically sported better system integration than Unix, with system components of greater granularity and flexibility than Unix has historically provided thanks to the power of COM interfaces. If the tightly coupled Windows way provides a better solution, then maybe we should build things the Windows way.

nothing that you said is actually true. There are probably 10,000 Unix systems for every windows system now, and all of them support far greater granularity and flexibility than any windows device has ever offered.

Making init better is a good thing. Breaking working systems is a bad thing. Making gnome not work on *BSD is a terrible thing, not just on a theoretical level but also on a practical one.

Linux itself would have been impossible if the systemd attitude had prevailed among older unix. Linux-only software is the open source community shooting itself in the foot. Systemd will break the community, which will be worse for everyone, linux users included.

Does anyone really care, though?

This is the free market deciding that non-Linux unixes aren't worth anyone's time. If you believe in the free market, you should either stop complaining and get on the winning team, or hole up in BSD-land rewriting Linux software until you have a plurality market share.

Demanding that open-source developers donate their time to support your particular OS is just leech behavior. Mandating "support equality" is a step in the wrong direction. Open source became great by being free; let's not take that away.

Is this the religious argument for systemd?

This isn't an argument for systemd at all; it's an argument against supporting any OS with more than one user.

If your OS doesn't matter enough to demand support, it's well within my rights as a developer to ignore you and your users entirely. Cowtowing to a hoard of complaining users is weak behavior.

> This is the free market deciding that non-Linux unixes aren't worth anyone's time. If you believe in the free market, you should either stop complaining and get on the winning team, or hole up in BSD-land rewriting Linux software until you have a plurality market share.

If you believe that then why use Linux at all, when windows' market share is so much bigger? And I thought boycotts were supposed to be the free-market way of changing company policy.

> Demanding that open-source developers donate their time to support your particular OS is just leech behavior. Mandating "support equality" is a step in the wrong direction.

Not supporting *BSD themselves is one thing, but systemd explicitly refuses patches to make it more portable.

>And I thought boycotts were supposed to be the free-market way of changing company policy.

Boycotts are what shrill activists on the losing side engage with. The free-market way to change company policy is to start a new company and make new policy. Doing otherwise is just trying to leech off of the success of other, stronger people.

>Not supporting *BSD themselves is one thing, but systemd explicitly refuses patches to make it more portable.

This is exactly what OpenBSD is doing with LibreSSL, so excuse me if I don't listen to the pot calling the kettle black.

The fact of the matter is that Linux won. Systemd won. Alternatives had every chance to make themselves successful, but they lost, and they have nobody to blame for that but themselves.

good point.

I don't see init scripts as an argument for writing journal (no more plaintext logfiles), integrating udev, replacing cron, writing their own login manager or handling power management (among other things).

The argument of "do one thing well" might be 20 years old, but no one is holding progress back -- rather raising voices against doing so by aggressively removing choice from the agenda with it's current implementation.

This is all stuff that needs to be taken care of, on every system. So I'm happy that systemd handles the basics in a sensible way. Why do I as a sysadmin need the choice here? Everything I need is covered by systemd, and in a less painful and more consistent way.

For example, I really do like how systemd-journald takes care of logging, although I don't know why it's not using plaintext log files internally. journald gives me the choice to not log to disk at all, which is nice on embedded systems and my notebook SSD. If i wanted to have logs the old way, i'd just attach a syslogd on journald's socket. With systemd and journald I can have my services log to stderr and everything just works.

I don't have to: * Log to stderr if my service can't start for some reason * THEN daemonize (which is a pain to do in node.js and python, and not possible in golang) * Then switch logging to syslog or my own logfile(s) * Think about logfile rotation * use supervisor or monit to make sure my services stay up

Sure, it needs to be taken care of and the creator of that page recognize that when they say "We do recognize the need for a new init system in the 21st century". Systemd is trying to do TOO MUCH, instead of being a part of a larger system it's trying to become the system. Ad I don't like it, because I don't like tight coupling.

PS: as an alternative you could use a tmpfs to move /var/log (and other log directories) to RAM on an SSD system. Far less configuration required, in my experience. And it works with every *nix out there.

That is a terrible alternative. It is a colossal pain chasing down processes which are still doing open-ended logging into /var/log on a RAM-only system, or guessing which ones may or may not be properly rotating their log files.

The OP is right: some stuff is so essential it should be centralized. Whether systemd is the right way to do that I don't know, but it certainly seems to the compromise we'll live with in the near future. Just as SysV was in the past.

> Even if it's technically wrong: Unix as a platform is now Linux. Everything else is niche products, and we don't have to use the same init mechanism on Linux/BSD/whatever just because we can.

I think it's a shame most of the original page comes off as backwards thinking, Unix nostalgia and FUD like claims.

Because this one thing is actually quite important and something we should care about.

You can claim that Linux is now Unix and everything else is just niches. Great. Like Hurd. Clearly a play-thing. No point being portable enough to work there.

However back in the days, Unix was the real thing and Minix/Linux was just a toy itself. But hadn't software initially been portable, Linux could never have risen to what it is today.

When we are making an init-system with Linux kernel-dependencies, and making other core platform services like DEs dependent on this init-system again, we are tightening the bonds beyond what's useful, effectively hindering other platforms from ever gaining mass adaptation, even if they should otherwise deserve it.

SO while I appreciate systemd in general there are aspects of this development I indeed thinks deserves some more and warranted criticism.

> How could this get even worse?

Linux distros have gotten worse and worse and worse over the years. (So has Windows.) Every 3 years the old way of handling configurations is thrown out and a new one put in place.

That said, I don't know enough to say whether systemd is fixing this problem or if it is contributing to this problem (which it could do even while claiming to fix the problem).

If you can't access the site, your toughs about it's current content, look like a you're respectfully trying everybody to disrespect this.

I'm happy about this movement.

In 15 years of sysadmin, I can say that writing init scripts never was a big issue. Maybe for "junior sysadmis" or those that are in the office by their last name.

I respect your opinion, but I'm very afraid of the course that linux is taking regarding system design (dbus, systemd, etc) and I resonate with that page content.

I read your answer like this:

No newbie sysadmin should touch init scripts because they can't handle it.

Isn't it about damn time we go from overly complex to what makes sense?

“The only way to control chaos and complexity is to give up some of that control”

>No newbie sysadmin should touch init scripts because they can't handle it.

There's a lot of things newbie sysadmins are going to be bad at. I wouldn't expect them to come in and architect an infrastructure that will be handling millions of events per second, either. Nor would I expect them to be able to perform a deep dive and become performance engineers.

But sysvinit doesn't have to be the answer. I'm an Illumos guy. The SMF implementation is excellent, moves away from init scripts, and still doesn't throw the baby out with the bathwater like systemd does.

If I was interested in giving up control for the sake of reducing complexity, I'd go back to being a Windows sysadmin so I can click next on wizards and run dcpromo all day.

Without taking a position on the larger issue, with init scripts, another sysadmin could always figure out what the junior sysadmin did.

Jr: "I edited some init file."

Sr: "Which one?"

Jr: "I forget."

Sr: "Oh." sorts /etc directory by time "Oh, I see what you changed. That is wrong because of $REASON. You should do $CORRECT_THING instead."

Jr: "Okay, thanks for letting me know."

Sr: "We were all new guys once."

You can do that with systemd too -- service files are still just ordinary files; they're just "10 lines of info that matters" instead of "10 lines of info that matters + 90 lines of inconsistently implemented boilerplate" :P

More like "10 lines of info that matters + I don't know/no reason to look at that" instead of "10 lines of info that matters + 90 lines of inconsistently implemented boilerplate"

Saying that systemd service files are complicated because you need to take systemd's internals into account is like saying sysv init scripts are complicated because you need to look at the source code to "cat" and "grep"; ie, no :P

In 20 years of managing Linux systems, I can probably count on one hand the time I saw an init script that was written properly unless it was provided by the distro maintainers (and often not then either). That is, it included all the expected options for the service, it correctly handled pid files for daemons that for whatever reason doesn't do it themselves, it handled reload's gracefully, and so on.

Good riddance to init files.

If these people want to keep people from using systemd, they need to come up with a better alternative.