Hacker News Reader

Comment by bitwize | Hacker News Reader

Nice article, but at this point it's game, set, match -- systemd has won. The Linux community overwhelmingly supports it, and its advantages clearly outweigh its perceived disadvantages.

"It's not Unix" is no longer a valid critique. The old Unix design philosophy is dead. People want systems of components that function as an integrated whole. They do NOT want pieces that are generally unaware of each other (this is touted as the virtue of "loose coupling").

Yes, pid 1 does a lot. That's by necessity. It's not your grandpap's Unix anymore. Do you want to control all the system services, including the "badly behaved" ones? That functionality needs to be in pid 1. Do you want to bring the system up safely and sanely, accounting for things like hotplugged disks in /etc/fstab? That functionality needs to be in pid 1. Systemd is the correct design for modern Linux. It also brings the advantage of better integration and greater commonality across distros.

View on HN

Even on systemd that stuff isn't all handled in pid 1.

There are 6 systemd service processes running on my machine handling different components.

Snipped out the irrelevant processes.

    % systemd-cgls
    ├─user.slice
    │ └─user-1000.slice
    │   └─user@1000.service
    │     ├─6612 /usr/lib/systemd/systemd --user
    │     └─6616 (sd-pam)
    └─system.slice
      ├─1 /sbin/init
      ├─systemd-udevd.service
      │ └─151 /usr/lib/systemd/systemd-udevd
      ├─systemd-logind.service
      │ └─345 /usr/lib/systemd/systemd-logind
      └─systemd-journald.service
        └─125 /usr/lib/systemd/systemd-journald

People want systems of components that function as an integrated whole. They do NOT want pieces that are generally unaware of each other (this is touted as the virtue of "loose coupling").

This may be true of "mainstream" users' applications, but is it really true of system software? And should we really attribute the cultural shift toward monolithic systems to shifting preferences rather than leap-first "design"?

Why do those things need to be on pid 1? Just asserting it is not really convincing.

Any init system needs to be pid 1 [1]

This blog post does not identify the subtle argument that upstart maintainers made against systemd at pid -1 : it also brings along with all the other services (like logind) that frameworks like Gnome depend on.

So, if Gnome wants to depend on logind and logind depends on systemd (post systemd rev-205), this means that Gnome is irrevocably coupled to systemd which must run as PID-1 [2]. This means it becomes hard for upstart to be used with Gnome.

There has been no doubt whatsover that an init system needs to run as PID-1.

[1] http://en.wikipedia.org/wiki/Process_identifier#Unix-like [2] I must add here that Gnome does include a fallback path to not depend on systemd components.

A part of any init system needs to be pid 1.

The question is how much of it, and whether or not Systemd piles too much stuff into pid 1 that could instead be farmed out to separate processes, handled by separate binaries, that could be more easily upgraded or replaced.

(I don't know; I've not looked at Systemd enough to have an opinion)

they are separate binaries - however all of them depend on systemd (post rev 205) and cannot be run independently.

this is because all of systemd's services rely strongly on cgroups for resource control and upstream kernel wants a single reader/writer for the cgroups API (the current filesystem model is going away). systemd is just following lockstep with what kernel wants and is becoming that API.

There has been a lot of FUD around the "PID 1" disaster, but invariably every init system will have to have a single point cgroup writer. Systemd is just ahead of the curve.

http://lists.freedesktop.org/archives/systemd-devel/2013-Jun...

Anyone has more information about this from the kernel perspective ? Will systemd be the only API ? Sounds strange.

This is a reasonable evaluation of the status - https://lwn.net/Articles/575672/

Basically, if you use systemd as the init system, then yes - it is the only API [1] . If you dont want to use systemd then there are alternate implementations of the cgroup API (e.g. cgmanager - http://cgmanager.linuxcontainers.org/ ). IMHO, they are both incompatible with each other.

Lennart has a reasonable justification of why systemd is implementing it's own API [2]

[1] https://lwn.net/Articles/557111/ [2] https://lwn.net/Articles/557140/

PID1 also comes with powers over capabilities and (in some cases) cgroups. You can do much of that with FSCaps, etc - but to control the bounding set, you have to be pid1. So - if you want init to be able to limit what processes can do, you will be pid1.

Linux has always supported userspace diversity, so nothing wins completely.

We are at a point where containers are going to change what people want in a Linux distro and not sure that monolithic will win.

Containerization will make this entire kerfuffle pointless. Running single image big machines with lots of services is a 90s and early 2000s architecture aesthetic that will fade into the background as individual services migrate to a PaaS model for enhanced modularity and mobility.

I haven't looked deeply into systemd, but from how the conversation around it is going, it would seem to be solving complexity with more complexity.

I'd like to see a mashup of CoreOS plus NixOS integrated with a cluster container management system.

"is a 90s and early 2000s architecture aesthetic"

I hope you realize "containers" are so similar (identical compared to "single image big machines") to how mainframes worked in 70's.

Just saying nothing is new, cycles are the norm and don't be so high and superior cause your new hotness is gonna be the "old, broken shit" in a couple decades.

And mobile is the new desktop because there is enough computational power there. Soon, we will see containerization on mobile because of too much complexity and not enough security.

Containerization has always been a good idea. http://en.wikipedia.org/wiki/FreeBSD_jail has been around for almost 14 years. I am aware that full system virtualization dates back to the 1960s, in no way do I think that old eq bad.

CoreOS is of course systemd based...

I know, but CoreOS won't be the only lightweight OS designed for cloud deployments. I'd be pretty jazzed for a Lua or Gambit Scheme based system. It will probably be node.js though, always bet on javascript.

Been working on Lua based stuff that can bring up enough of a system for containers (networking and so on, still some bits missing) https://github.com/justincormack/ljsyscall

You mean L33tStart? :)

http://www.reddit.com/r/programming/comments/14ay0r/hacker_k...

I wish l33tstart was a real thing. Maybe have it talk over nanomsg using capnproto? But the messages should be cryptographically signed. Would it handle logging as well? Could those log messages get processed through summingbird or storm? Ok, Clojurescript -> Gambit Scheme -> Native exe

daemontools doesn't run as pid 1 and achieves all of this.

http://cr.yp.to/daemontools.html

It does not. Not even close. https://news.ycombinator.com/item?id=7210308

That comment is insane. Why would log files need journaling? They should be serialized.

How is adding a line to a run script "duplicating logic badly"?

Coupling is clearly bad, as evidenced by ... 30+ years of software design, but is a necessary evil. I don't see any reason why it's necessary in such a low level system component. "this is not rocket science" you don't impose dependencies on something you want to be reliable.

Daemontools is clearly the most mature solution, from a perspective of stability. It's probably why nobody wants to use it, there's nothing to play with or exploit for profit because it already works.

That comment is insane. Why would log files need journaling? They should be serialized.

systemd-journald provides tampering-attestation features that plain-text logs do not, and cannot, provide. It's fundamentally more secure.

Fallacy: having owned the user that writes logs, I cannot fuck with your logs if journaled.

Reality: if I owned the user writing logs, I can do anything I want with the logs, no matter how they're stored.

No, you cannot do anything you want with the logs.

Here is something you cannot do: you cannot alter the logs without anyone noticing, since log entries have a rolling hash.

This prevents an attacker from being able to modify logs without detection, and was implemented in systemd after the postmortem on the kernel.org breakin, i.e., it was developed to counter an actual threat.

I would suggest you read up on what log attestation actually does.

Logs can be modified in real time by being intercepted during writes to the buffer before the new hash is applied. Because of this, only the logs up to the intrusion can be preserved, and even then if the old message/hash/key/etc is still in memory, the old messages can be modified.

The only truly secure way to record messages up to the point of break-in is a one-way remote log. Any new messages after break-in is subject to falsification.

> The only truly secure way to record messages up to the point of break-in is a one-way remote log.

And if you have this then you don't need a fancy cryptographic syslogd to detect tampering.

The fancy cryptologic syslogd is fallible as I mentioned previously, so why depend on it? And really if you really want to cover your tracks you can just cause random disk corruption over the whole disk (and just happen to damage the journal in the process). Things will start failing rapidly, fsck will later show itself fixing the disk corruption, and it will be assumed to be a hardware error and the incident ignored.

Security half-measures do not mean you are secure. If it can be hacked, it will be hacked, and then what was the point of sacrificing your whole system's init system?

(Also you could just replace syslogd with a journaled syslogd, rather than replacing your entire init system... but I guess that's off-topic?)

I'm agreeing with you.

>rolling hash

FAIL. You just wrote outdated software. Under this dumbass idea of journaling, which by defition is there to allow editing.., the best thing you can do is built a rolling function, which basically means coping the data, which basically means increasing your attack surface.

From what I understand systemd can optionally store a copy of the logs on a remote server. The key to modify the logs on the remote server is changed in such a way that even if a system is compromised the log copy on the remote system can not be changed.

These remote copies of the logs could actually be used to detect log-tampering and 0day exploits.

>From what I understand systemd can optionally store a copy of the logs on a remote server.

anything can have any feature, what matters is how it's built.

The journal sealing provides forward security. The sealing keys are exchanged regularly, and the old keys are deleted safely. A verification key, which is stored on another device or offline, can be used to calculate the sealing key for any given moment. While an attacker can change the logs, this will always be visible in an audit.

http://lwn.net/Articles/512895/

https://eprint.iacr.org/2013/397.pdf

the old keys are deleted safely

Assuming nothing has managed to redirect those unlinks for example...

I guess you're right. The mechanism is forward-secure, which means for an attacker that tampering the logs is the first thing to do, not the last ;-)

As soon as the attacker manages to extract the state of the key generator from memory at one point of time, the log entries from that and the following sealing periods can be modified and cleanly sealed.

rsyslog allows you to sign your logs if you want to: http://www.rsyslog.com/how-to-sign-log-messages-through-sign...

How does svscan handle service dependencies?

It doesn't. That's the point. Nothing in daemontools does. Dependencies are bad.

I'm not sure if you're making a point more subtle than I'm reading, but when most people in the systemd debate say "dependencies are bad", they mean that having loads of things depend on the init system is bad.

The comment you're responding to is talking about dependencies in boot services, e.g., sshd depends on having the network interface up. Systemd does a great job of handling this sort of thing, allowing services to start in parallel, but ensuring that things only start after all the pieces they rely on have started. That sort of dependency isn't "bad" -- it just is. You can't remove a dependency like this, it reflects reality.

That said, I really don't care for systemd. I think the complexity it adds more than counteracts the benefits it provides, but I'm old and didn't mind SysV or BSD init.

I'm saying that the dependency on the network being up is bad for services. An example of a bad case: if the network interface is hotplugged or down and you're running Redis, do you want it to restart and have to drag the entire AOF file off disk just because you yanked the interface card because it was failed or failing? Your 20 second hotplug problem becomes a 15 minute block of IO coming off disk because you had to restart the process.

So many problems come from such a simple dependency.

Designing all services to handle and expect failures and just let init restart regardless of the state is the RIGHT solution.

daemontools can't reliably monitor daemons that reparent themselves to init or kill their pgroup.

I have written hacks to work around this (I have a gentoo VM that uses supervise as an openRC replacement, and have several hacks to work around this issue). It's a pain in the ass, and one reason why systemd runs in pid 1.

I prefer daemontools, but it's not perfect; it made a different set of tradeoffs.

Linux was never UNIX. Influenced yes, but not UNIX. That said, in the FOSS world it is never 'game, set, match' it is 'game, set, fork' :-). For what its worth though, these sorts of debates have happened forever, MOTIF vs CDE, Streams vs Sockets, Signals vs Events, TCP vs OSI, Etc.

I'm not so sure this is entirely true, especially with pid namespaces becoming a real thing. The true pid1 could be greatly simplified for the sake of system stability while providing sub-pid1s that handle actual service, device, and application management.

Everyhing in pidns is destroyed when pidns init dies.

I'm not sure what you're getting at here. Yes, that is obviously true. As is the case with the true pid1 init. Care to expand?

I mean if you protecting yourself from panic by moving stuff from real PID 1 into pidns root, you're doing it wrong. pidns root will segfault and kernel will SIGKILL everything inside which is equivalent to good old panic.

> systemd has won

Not in the Gentoo ecosystem where we have OpenRC: https://en.wikipedia.org/wiki/OpenRC

Gentoo is still popular as it's the basis for ChromeOS and CoreOS, but CoreOS uses systemd and I'm not sure re: ChromeOS.

I was under the impression ChromeOS used upstart.

ChromeOS uses upstart.

Also Debian is still on sysvinit, and there is a vote being held to decide what init system to use for jessie as default.

That vote has almost total agreement that systemd will be the new init system. At this point, they're stuck in a quagmire of other bureaucratic nonsense and debates over related issues like whether packages will be allowed to require a particular init system.

Or maybe they need to figure out how to keep the non-Linux ports working since systemd is highly tied to that particular kernel.

neither the hurd nor the bsd port use the debian linux sysinit currently.

We'll likely switch to systemd, over at Gentoo, when we need to. As we discussed at FOSDEM, it's a matter of 'when' rather than 'if'.

If that switch happens and systemd is forced on us I'll be dumping Gentoo after 12 years.

systemd gives me nothing that openrc doesn't already provide and it messes with a lot of things that I do care about (eg. logging, cron).

If RedHat is so intent on forcing their junk through everyone's throat it's time to move back to an OS where technical insight reigns instead of corporate money and marketingspeak.

To be honest, if Gentoo moves to systemd it'll probably be the final straw that pushes me to switch back to Windows after a decade or so of Linux use. Things have just being getting more and more buggy and unreliable; I'm already on the verge of switching just so I don't have to deal with the state of audio on Linux these days. Every PulseAudio version introduces new and fun breakages and they don't do stable releases anymore, so at best it'll get fixed in the next release which will have a bunch more poorly-tested code that breaks something else.

I have been able to keep PulseAudio out of all my Gentoo systems so far. I seriously doubt anybody can impose systemd on me.

If this were true, why would you bother saying it.

Because the holdouts in the init wars debate are whiners who are trying to obstruct useful progress being made on the Linux platform, e.g., Ian Jackson.

Reply on news.ycombinator.com