Hacker News Reader

Comment by atonse | Hacker News Reader

I get the impression that OP seems to just be abandoning systemd to build another... systemd. So it seems like not-invented-here syndrome.

I don't really know about any of the drama surrounding systemd but I've liked using it so far. It's easily the most attractive feature in Ubuntu 16.04 LTS.

Using journalctl for logs is nice because I don't have to keep looking around for where logs are stored and then tail them. But then again, I don't worry about log rotation and things like that. My guess is that journalctl takes care of it.

I'm sure I've barely scratched the surface of all the things it can do (like auto-restarting daemons, some security stuff, etc).

Am I the exception? Can't be so, if most linux distros have adopted systemd.

View on HN

I'm sure there's room for reinventing a better systemd, but most of the attempts I've seen either fail to understand and intentionally ignore just how many things systemd simplify. I can understand wanting to decouple and simplify that, but I think a lot of the time the people trying to reinvent it do not understand the concerns that systemd solves for people.

E.g. a typical issue is process restarting. Simple. Can be done in a couple of dozen lines. Except that solves nothing for most people: If your server is spinning in place trying to start your big-proper-process-monitor, and it's failing to log anything because rsyslogd or whatever is also just spinning in place, and the boot processes that should have mounted the filesystems needed to get things working is repeatedly crashing?

You can mitigate all of that, but now you're creating a separate mechanism (maybe as simple as writing to disk, but still) for early-stage logging, and a separate mechanism and your init needs to deal with things like throttling restart and handling them and so does a separate process manager, and so on.

In order to simplify things, there's duplication of functionality all over the place.

A huge part of the architecture of systemd is down to starting to pull at that thread and untangling all of these issues. Is it perfect? No. Is it the solution to all of the concerns they've uncovered? Probably not. But what we had before was so much worse, and I'm so happy someone have had a go at addressing them.

And frankly, it's great that animosity towards systemd has motivated so many people to finally try to address this issues in other ways. Maybe one of them eventually solves what none of them did before systemd forced peoples hand.

I think for many people who are dissatisfied or angry, systemd didn't simplify things. Many people have simple use cases, and for them it made things much more complicated. Monolithic, incomprehensible, unintrospectable. It is like a modern car engine which is just a sealed black box under the hood, full of electronics. Sure, it works great, it is progress compared to before, everything is there for a reason. But you can't easily fix things yourself. And for many people, the old kind of engine was good enough.

I mean, for my simple uses, what is wrong with `sudo service nginx restart`, or `sudo killall nginx`? For many people, it is really dead simple. I get that it is not simple for everybody, but that is why we should have choice, not one-size-fits-all.

> I mean, for my simple uses, what is wrong with `sudo service nginx restart`

Not sure, but I'm using SystemD and do this literal command nearly daily. What exactly is your point here?

Good point, on Fedora 26 I see /usr/sbin/service is provided by initscripts-9.72-1.fc26.x86_64 and it's just translating the service commands into systemd equivalents.

    [chris@f26h ~]$ sudo service smartd.service status
    Redirecting to /bin/systemctl status smartd.service

An excellent summary. To me it flies in the face of UNIX' killer feature, it's simplicity. That it seems to be one guy pushing this barrow is worrying.

I don't think there's `one guy pushing this barrow' primarily because everything but Gentoo uses it. (And event then, you _can_ use it).

One company actually, called RedHat.

Canonical fought back for a while with upstart, but they lost.

See, RedHat got their assorted junk integrated tightly into GNOME (partly also KDE) and "legacy" options killed.

    It is like a modern car engine which is just a sealed black box under the hood, full of electronics.

One of those fully open source sealed black boxes?

    I mean, for my simple uses, what is wrong with `sudo service nginx restart`, or `sudo killall nginx`?

How is 'systemctl restart nginx' any harder? service is specified as 'Run System V Init Script' specifically in the man page, and there was a transition period where some scripts still were in /etc/init.d before all of them had been converted to systemd units, so at times both service and systemctl would have been present and doing slightly different things. So it makes a lot of sense that they didn't give systemctl the same name (see https://linux.die.net/man/8/service).

Why do you think killall wouldn't work? It's present by default on my Fedora machine here...

> One of those fully open source sealed black boxes?

That's a fallacy. Webkit is open source, doesn't mean it is not completely opaque to the majority of developers. I'm not concerned about the legal free/open source status here, but about how well a casual user can grasp and change things (that is someone who has to start, stop, maybe install services, and isn't a systemd developer).

I'm not bothered that the name of the tool is now `systemctl` of course. My problem is that is does way to many things. Want to list services? It shows you hundreds of pseudo-service-objects. They can get activated at any time by mysterious means. If you need this, it's great that it is available, but for me, the old ways (as many failings that they had) were good enough.

All of those dependencies were there before too. You just didn't have nearly as good visibility into them.

> But you can't easily fix things yourself. And for many people, the old kind of engine was good enough.

I'm not sure that's a good argument. If you have a medical problem, do you try to use it yourself?

Do you wish that you were able to fix your computer CPU when it breaks or do you just buy another? If you were to make your car/computer more fixable, it would probably be more expensive/less reliable/...

an init system is software and we're software people. It's not at all like a non doctor making decisions about medications.

I don't understand the difference between 'sudo service nginx restart' and 'sudo systemctl restart nginx' and also 'sudo killall nginx' still works with systemd. So I'm not understanding the simple use example.

Whether SysVinit scripts or systemd scripts, you have to learn something to do either one. I just see it as a difference rather than which is simpler. But systemd can use SysVinit scripts anyway if you prefer it.

The latter is a lie for more advanced scripts like what OpenRC provides.

I don't think that is a simple use case of an init system; using pre-built init scripts is the default use case. A simple use case is writing your own simple startup scripts for your own services. For that case, systemd is very, very easy to use

I find the "monolithic, incomprehensible, unintrospectable" complaint particularly odd. It has made it so much easier to slice and dice logs, make sure you have logs in almost any conditions, massively increase the ability to request information about running processes, and how they are related etc. "systemctl status [service]" alone is sufficient justification for system. Being able to get resource usage per service instead of just per process with "systemd-cgtop" too. "systemd-analyze" is also fantastic (try "systemd-analyze time" and "systemd-analyze blame" and "systemd-analyze critical-chain"). Systemd have made it vastly easier to learn about the boot process without being forced into intrumenting dozens of shell scripts.

As for simple incantations, "sudo service nginx restart" works just fine on Debian with systemd. It's trivial to provide wrappers. But not exactly hard to learn "sudo systemctl restart nginx" as an alternative. There is a small but real learning curve to replace your old habits, yes.

As for "sudo killall nginx" it is indeed easy, but it's also an awful habit. Suddenly you're on a server with multiple different nginx instances (in different containers, perhaps; been there, killed those by accident; stopped using killall). Or you're applying killall to a process that will leave a mess. Or you have a server process that spawns children that double-fork and won't die just because you've killed the server.

That simple case is indeed simple. Until it isn't. And that's one of the many caveats where systemd goes far further than most of the alternatives by e.g. using cgroups to keep track of related processes even when they do stupid stuff to confuse you. In return you get to do "sudo systemctl stop [unit]" and know that not only will it be stopped properly - along with all related processes, but any other dependencies will get shut down cleanly.

We do have choice, but in most cases where the "dead simple" cases matter, it is a matter of relearning a handful of commands (with wrappers for some helpfully provided by some distros) for the dead simple cases to work, with the bonus that all kinds of ugly corner cases most people don't even think about just stops being a concern.

If your needs are dead simple, then systemd is dead simple too. When your needs are not dead simple, most of the alternatives can't solve them properly, and you often end up having to cobble together awful monstrosities. Or trying to figure out what in the world went wrong.

The basic summary is basically: Developers love systemd, it's much more "developer friendly" and the way it's laid out makes sense to them (e.g. your love of journalctl, which is bizzaro-world for me). No more thinking about ugly operating systems and those silly details!

Old school sysadmins and those who actually get their hands dirty with Linux systems under the hood despise it for what it is. It's the windowsfication of Linux, in my - and many others - opinions. A giant PID 1 that controls everything, re-implementing such basics as ntp and rm, is imo entirely anti-unix-like.

Systemd is great until it breaks. Once it breaks though, good luck. They've re-invented the wheel multiple times over, and as time goes on more and more things become part of monolithic systemd. If you do happen to find a weird bug, prepare to get yelled at and be forced to vigorously defend your bug report. The joke in the office is when Pulseaudio support gets added.

> A giant PID 1 that controls everything

Kernel is also a giant code that controls everything. Is this such a bad thing?

Yes. There's been a lot of theoretical and practical work around the last couple decades about putting as little as possible in your single point of technical and legal/patent failure.

Small tools that do as little as possible, but do it very well, composed into bigger systems.

Philosophically identical to RISC vs CISC.

The kernel is a component that has a straightforward purpose on a system, and interferes as little as possible with the actual usage of the system. You could describe it as "It gets the job done, and stays the hell out of the way"

Even with the kernel there is an element of choice about what parts of it you do / don't use, e.g. you can use ext4, XFS etc. etc. and even have the choice of providing kernel modules so you can bolt on ZFS.

The kernel doesn't start to dictate who or what software does DNS resolution, for example.

systemd keeps taking over more and more control of various aspects of your system and effectively removing choice by making it hard to use alternatives because it starts to make assumptions based on the inbuilt versions.

systemd is presenting you with a "one-size-fits-all" scenario. The problem is one size never fits all.

One of the biggest strengths within the *nix ecosystem has been the fundamental understanding no two use cases are entirely the same, what works well for one purpose doesn't work well for others, and choosing the right software to meet those various purposes is invaluable. It's impossible to make software that is efficient for every single use case.

The freedom and possibility to choose the right tool for the right task is essential to designing and building new, innovative, or even just plain old efficient systems.

I hear all this, but I never really see examples of it, which would really help your argument. People hear the 'theoretical', but theoretical doesn't matter if it never makes it to the practical.

Okay, so an example:

dnsmas

"The problem is one size never fits all."

Well written and agreed with but I'd add management of legal risk and labor productivity.

You could compile a http server into the kernel, which would benefit microscopically in ways almost all people don't care about, then product tie it into other projects to successfully kill nginx and apache off.

Or you could simply run Apache in a non-monolithic way.

What happens if a submarine patent surfaces? Or a critical "must fix this instant" security bug hits?

Well, if you're running Apache and not doing anything too weird, you'll be back up and running on nginx in a couple minutes and the world yawns. Or you can upgrade/downgrade/patch apache without messing up your DNS service or NTP service or whatever else.

But what happens on a monolith? Well, you could ... I guess you decide if you're better off infringing the patent or replacing the ENTIRE monolith.

Those things happen. Eventually its inevitable that something in the monolith will become unacceptable to run, legally or from a security perspective, and then the entire monolith has to go, simultaneously. Life with separate tools is incredibly easier.

Embrace, extend, submarine patent, extinguish.

There are also labor productivity issues with monoliths. Best expressed via a standard HN automobile analogy, where technically I could completely restore a vintage jeep automobile using one tool, a swiss army knife. Simplistically, all automobile restorations should be done with swiss army knives to minimize the paradox of choice and minimize the massive expense of tool training. In practice it seems obvious I would infinitely prefer not restore an automobile using a single swiss army knife as my only tool. Even a really bad toolbox would be better than using one swiss army knife. Given the choice between using an old, slight imperfect yet historically proven highly successful toolbox vs only being able to use one brand new untested swiss army knife... thats the kind of thing that makes me glad I use *bsd instead of linux.

> Even with the kernel there is an element of choice about what parts of it you do / don't use, e.g. you can use ext4, XFS etc. etc. and even have the choice of providing kernel modules so you can bolt on ZFS.

You can also do the same with systemd. Don't want to use journalctl? Fine, forward logs to syslog. You can argue that you can't disable journalctl, however depending of the feature you want in kernel you can't disable some modules thanks to inter-dependency.

> The kernel doesn't start to dictate who or what software does DNS resolution, for example.

Neither does systemd. I don't use systemd-networkd at all in my machine and it still works fine.

> systemd keeps taking over more and more control of various aspects of your system and effectively removing choice by making it hard to use alternatives because it starts to make assumptions based on the inbuilt versions.

I kinda concur with you, however I also try to understand why they're adding support for all those things. In general, there is a sound resolution to include something in systemd, even if I or you does not agree.

And remember, except for journalctl, systemd does not force anything at you if you just want its init system.

> systemd is presenting you with a "one-size-fits-all" scenario. The problem is one size never fits all.

It is not. All systemd implements are foundations to build bigger and more advanced functionality. systemd-nspawn is basically chmod on steroids, you can't call it a container system. However, you can use systemd-nspawn to build a container system like rkt. systemd-networkd is also too bare bones to be called a "network manager", etc etc.

> One of the biggest strengths within the *nix ecosystem has been the fundamental understanding no two use cases are entirely the same, what works well for one purpose doesn't work well for others, and choosing the right software to meet those various purposes is invaluable. It's impossible to make software that is efficient for every single use case.

See above response.

> The freedom and possibility to choose the right tool for the right task is essential to designing and building new, innovative, or even just plain old efficient systems.

Yeah, that is why systemd does not force you to use anything (except journalctl).

Of course it is! Which is why it's preferable to put things in user space whenever possible. There were even microkernels for a while to try to mitigate the design difficulty of kernels; however at least with Linux what won out was lots of testing and very very picky maintainers.

Even if systemd is at the very high standard of kernel reliability, it would be better design to break out the monolithic parts where possible. The kernel is the way it is because it needs to be, not because it's a preferable way of writing code.

I see this kind of thing mentioned a lot which really disappoints me because systemd is not as monolithic as people seem to think it is. systemd (the init system) is a relatively small component with a modest set of dependencies. systemd (the umbrella project) includes a number of other projects which are separate and many of them aren't even required and are easy to replace with something else if it's needed.

If you're a programmer I highly recommend taking a quick browser through the systemd source code, it doesn't take long to see that it isn't really monolithic at all and is in fact a combination of individual tools which are combined to accomplish things. Just because they're all under the systemd umbrella and all in the same repository does not mean it's all one binary and that parts can't be swapped out.

That one can liken pid1 to the kernel should give one pause for thought, and not in systemd's favor...

I think this is very well put. The ever-going philosophical debate is about interface complexity (developer vs sysop), not how it actually works.

And also between small and large deployments.

Systemd has over the sprouted a whole bunch of parts that fit large (cloud) scale deployments.

Where things get messy is when all you want to do is to spin up a gray box on the shelf to handle some internal data, and at that point systemd, and a whole bunch of other "plumbing" that has sprouted within the Linux ecosystem in recent years (often under the Freedesktop umbrella, even though it seems to have a ill fit on desktops), becomes a beast to wrangle into submission rather than an assistant.

I, for one, am a old school sysop that is very happy with systemd as a modern service manager. I bet there are literally dozens of us :).

> A giant PID 1 that controls everything

> monolithic systemd

Technically speaking only small portion of systemd is running as pid 1. This has been explained many times: https://news.ycombinator.com/item?id=9515789 and I'm not going to explain this again.

> re-implementing such basics as ntp

Systemd-timesyncd is a minimal ntp client. This is not a replacement for ntp. You are free to use full blown ntpd if you want.

Think of it as a dns resolver inside your libc vs running proper caching dns server. It's the same situation.

> more and more things become part of monolithic systemd

https://www.youtube.com/watch?v=bdmv2FQRHWg :)

Well, not all developers. The first thing I do on systemd Linux systems is install rsyslogd and runit. We run our own services under runit which is started by systemd (because that's what the distro ships with). Runit works flawlessly.

systemd is not perfect. Debugging systemd is still very hard. There is no way to simulate what actions would be performed to reach a target without actually executing them. There are plenty of issues related to boot and shutdown which are _hard_ to track down because of this.

The journal is nice, especially on desktop systems, but it's way too slow to be used in any moderately busy system. The disk space utilization is also _prohibitive_ for archival.

We still rely on remote syslog because of this, but it's not without issues: the journal->syslog translation is lossy, and it's compounded with some "quirky" upstream decisions (purely cosmetic) to write unit descriptions in the log as opposed to the real unit name which make tracking down units harder if you don't have the actual journal file.

Journal files also get corrupted easily with unclean shutdown. In case of issues (from software to hardware), the local journal quickly becomes completely useless.

All in all, systemd is an improvement, but it lacks the debugging tools, performance and overall polish you would expect from a central piece of software such as init.

The internal APIs are mostly undocumented or self-referential. The DBus interface is opaque. Things gets muddy very fast as soon as you touch non-central components such as systemd-resolved or networkd.

I'm not a detractor, but there are many things systemd can still improve, but it feels we're kind of stuck. I'm quite happy if we have some competition here.

> Journal files also get corrupted easily with unclean shutdown.

Huh? I thought that they introduced this weird binary log format precisely to avoid this kind of error. (I don't remember the exact reasoning, though.)

You're confusing the binary format with the tamper protection, or file "sealing". The two features are really independent.

"They" were not very successful in this, like they weren't in any other stability related feature.

The journal files can get corrupted, but it's a localized corruption that journalctl skips over and continues on without additional data loss. Obviously anything that didn't completely commit is gone, which would be gone anyway with some other format. The main difference is that systemd-journald can a.) become aware of the corruption, b.) warn of it rather than display corrupt info, and c.) recover and continue on.

There have been a pile of bugs I've run into and reported where journald gets really confused; e.g. 'journalctl -b-5' won't actually go back five boots, it'll show some other content, or it'll give an error, but using --since will work fine. I haven't seen this problem recently, so the fix may be trickling out.

That's not my experience -- once corrupted it stays corrupted. Is this something that's been fixed since Ubuntu 16.04?

I don't know what you're contradicting because I never said the corruption gets fixed. I said it's skipped over. Specifically what happens is upon detection of corruption, journald stops writing to the corrupt journal file, creates a new journal file, and new entries go there. The utility for reading the journal files, journalctl, can also detect this corruption and will skip over detected corruption and continue on.

I have no idea what version of systemd Ubuntu 16.04 has, but in systemd-233 I'm not seeing aforementioned problems with -b switch and corrupt journal files.

I was building a system on raspberry pi involving a few daemons + things that needed starting on powerup + 24h after powerup + before powerdown.

It made all these things really easy, I'm not sure why everyone is so keen on doing everything with various bash scripts like we used to.

You are making the oft-made error of thinking that System 5 rc was the precursor to systemd. On several major systems it was not. The precursor was upstart on Fedora and Ubuntu, for example.

See http://uselessd.darknedgy.net/ProSystemdAntiSystemd/ for the System 5 rc fallacy and several others to avoid.

> [systemd] easily the most attractive feature in Ubuntu 16.04 LTS.

For me it is as well, I really miss it when working with a Ubuntu 14.04 box.

I have not found a single downside to systemd, really.

> Am I the exception? Can't be so, if most linux distros have adopted systemd.

By that logic why use Linux in the first place when Windows is so much more popular?

If it's anything like other projects from the same source systemd is great when it works and impossible to debug when it doesn't. Which is exactly what I came to OSS to get away from.

That's fine to me. Linux is about choice. One of my biggest problems with systemd is that other stuff starts to depend on it. For things like GNOME, it is not just an init system, it is THE one init system.

Also, I'm sure journald is useful, but I've had so many problems with it. For me as a Desktop and occasional remote computing, small server user, logs are ephemeral. Something goes wrong, I can quickly grep the logs which are plain text. I didn't really feel any advantage from the modern, binary, structured logs. Also, I spent hours installing something on WSL (Linux on Windows), finding that everything works except for the systemd logs, which are empty.

The command that controls services spits out hundreds of cryptic lines, services interweaved with all kinds of other objects, when I just want to list the running services. It has too many moving parts for my taste.

I don't need socket activation. I don't need a separate DNS resolver in my init system (unrelated note: I'd love more command line/GUI tools to be able to use a separate DNS resolver, work on separate interfaces, ...!), and so on.

Even if I will never use this guy's init system, if it is even a bit successful it will mean that people will stop assuming systemd, leading to a looser coupling, more interface stability, easier porting between systems, and more choice. So I think it's a great thing he is trying.

> I don't need socket activation. I don't need a separate DNS resolver in my init system

It's. Not. No idea why people keep repeating this. The DNS resolver belongs to systemd-resolved, is entirely optional and off by default. Nothing about it is in the init system.

And maybe you don't need socket activation, but there are plenty of valid use cases for it.

See: http://0pointer.de/blog/projects/socket-activation.html

"systemd" is the init system, therefore anything that has it in the name is part of that init system. See also journald, which nobody mentions as being part of the init system even though it's more tied to it (is it even possible to have systemd without journald?)

If they want to change that perception, they will need to change the name (and hopefully stick a stable ABI around it, but that's unlikely to ever happen).

Entirely "optional" yet it runs by default on Arch Linux Arm. I didn't even bother disabling it because it was a lot easier to install Void Linux instead.

The DNS resolver does not belong in the init system or part of it. You just need a working resolv.conf.

> abandoning systemd to build another... systemd

Well, systemd is a particular answer to a set of problems that have been well known for a while. Building an alternative solutions to those problems is not necessarily "another systemd".

> drama surrounding systemd

The quick summary: many people are critical of the architecture and design choices and implementation details in systemd in particular, rather than that the underlying issues it's supposed to address are real.

> but I've liked using [systemd] so far

This isn't surprising, as some of the problems it addresses are real, and a pain. But there are other implementations that also address many of the same issues. Whether those would make you as happy as systemd is another matter.

>I get the impression that OP seems to just be abandoning systemd to build another... systemd. So it seems like not-invented-here syndrome.

well, he does say:

>There may be a bit of NIH syndrome leading to this decision; that’s ok, I can live with that;

>I don't really know about any of the drama surrounding systemd

I have only been a passive observer of said drama, it boils down to A) init was a stable, widely used, and well understood system that didn't need an update B) systemd takes on far too much responsibility, contrary to the unix philosophy.

personally, I have experienced only minor annoyance when having to re-learn familiar commands, though there seems to be a init compatibility layer that has made it pretty transparent to me. It'd probably have been more annoying if I were still an admin.

> init was a stable, widely used, and well understood system that didn't need an update

If it didn't need an update why did various parties try and replace it e.g. OSX launchd, Solaris SMF, Ubuntu Upstart?

I have no idea. Like I said, I was only a casual observer of the rally against systemd. I get the feeling that the opposition was more about ethos than any irrefutable technical argument.

Init also took a VERY long time to start, and had very, very rudimentary concepts of dependencies.

Launchd is actually quite a hurdle to use. XML service files, inconsistent syntax for launchctl commands etc.

Apple probably designed it so .plist files would be editable with some GUI tool.

Reply on news.ycombinator.com