Hacker News Reader

Weren't Debian maintainers affected by the fact that it was... problematic to run Gnome without systemd? (Which might be entirety not systemd's fault.)

GNOME didn't have any significant systemd integration until well after Debian had decided to go ahead with it.

I remember this being a strong supporter, among others: https://lists.debian.org/debian-ctte/2014/01/msg00287.html

Then why did Josselin Mouette, a GNOME packager for Debian, write this in October 2013:

"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."

Correct. By the time the question of default init system was being decided by the Technical Committee, the tentacles of systemd were so entrenched that its relative merits or detriments as an init system were of little importance. As Russ Allbery (a committee member) pointed out at the time:

"We’re not really talking about choosing between two competing ecosystems. Rather, we’re talking about whether or not to swap out a core component of an existing integrated ecosystem with a component that we like better."

Also (to try holding back the tide of historical revisionism) let me highlight that Debian's decision was in no way unanimous and the process was not without its own controversy. In reality, the Tech Committee was divided down the middle when it came to their vote on the default init system, and the tie was only decided by the double vote of Bdale Garbee, the chairman of the committee.

udev was the biggest step towards "dynamic systems". Or should i say "hotplug" was, since it is more or less the same thing. I completely agree that the init should be an event based system. I'm not arguing that. I never argued that. I did not in fact argue much other then the statement that nobody should say their opinion about anything.

You say i left out stuff. I did. A lot more stuff then you wrote here. It would be a looooooong list of things, all of them simple (including "networking" and encrypted filesystems). I mentioned udev as an example of something slightly more complex, but even it is not.

> .. much better model than SysV init ..

You talk as a user, as in an admin. I talk as someone who was once interested in how all these stuff actually work. What i played with (read about, coded, and researched in general) includes init, udev, network hardware management (how ip and iw actually do stuff, excluding wpa), and more. What have i learned ? That almost all of it is done by the kernel. SystemD is not the only.. program that responds to events and goes over a graph and such. In fact, it is not even that hard to do what it does. Remember how i asked why udev doesn't have a generic event.. protocol ? You can scrap udev if you don't use systemD. And why ? For better integration ? The kernel tells udev when a device appears. So why doesn't udev just send a message of some sort ? It could even have a "mode" where it doesn't do anything unless it receives a message telling it to. So why not ? Oh, i don't think it's malice.

> The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.

The debian debate was a clusterfuck of grand proportions. There was one (1!) post on the whole debate that addressed the technical aspects. In the end the conclusion was that it is easier to just use systemD as stuff depends on it more and more (GNOME, notably, funny). RH is a huge company, they are not altruistic in their goals. I wouldn't say malice, but i wouldn't say innocence either.

And again, nothing of substance was said. Just fancy words and feelings. Brainwashing..

PS Fun fact: Xorg depends on udev to send it events over dbus when you plug something related to it. It still works without it, until you re-plug your mouse.

Wasn’t the debian vote stacked by a bunch of Canonical employees that voted on party lines, since Ubuntu had already decided on systemd?

I thought it was exactly the opposite, that Canonical employees were fighting for Upstart and Ubuntu only switched to systemd after losing the war in Debian-land.

There is no competitor because there was no problem. For most of use cases SysV, rc.d worked. Just like ifconfig.

Arch Linux switched to systemd earlier than Debian. And I have not noticed. Arch community accepts breaking changes. Debian? I believe they value stability.

Systemd should have matured, proved its citizenship and qualities. And it would have been accepted. Instead it usurped. People had no choice but to abandon their platforms. Decade of angry forum thread shows that there is a problem. Just like with python3, it's better to learn. And it is especially painful that there was PulseAudio just before. Same pointless change. Removed it and never looked back. ALSA works for me, still.

Usurped how? Who and how forced Debian and Arch to switch to systemd?

GNOME, essentially. If you want to run a fully-functioning GNOME on your distro, you have to be running systemd. Or have the time and patience to maintain a large patch set to work around the systemd dependencies.

This became this way only once systemd has won. At least during Debian systemd debate, both sysvinit and upstart could run Gnome just fine.

(also, what were the alternatives? I don't run GNOME myself, but given what GNOME wants do to, it does need a solid session management. I have seen ConsoleKit, it was worse than systemd.)

Debian's tech committee had their infamous vote about the default init system for Jessie in February 2014. Earlier, in October 2013, Josselin Mouette, a GNOME packager for Debian, wrote this:

"GNOME in jessie will need systemd as the init system to work with all its features, just like it needs the network configuration to be handled by NetworkManager."

You're missing a point. It's about feelings. Not my feelings at all. But empathy.

Replacement for "no problem" should run without problems.

python2 had no problems - there was a clear transition path:

    1. "str" => b"str"
    2. wait some time
    3. "str" is u"str"

ALSA had no problems - just wait till PulseAudio is ready.

SysV, rc.d had no problems - just wait till systmed is ready.

If SysV was as bad there would be same amount of outcry as with systmed

Was it really necessary to make it painful?

Who? Somehow systemd defenders has no empathy and Lennart Poettering is a prime example. No matter right them or wrong they are perceived as threat.

Just be nice. It may hinder innovation but it helps community.

Debian used to value stability. Nowadays - not so much.

> you left out networking and authentication, network service dependencies, filesystems, volume encryption, etc. – not to mention automatic restarts, conditional behavior based on the network or other devices (stuff needs to happen when I leave the office), and the various edge cases which I’ve seen in three decades running Unix systems.

I’ve been running Unix systems for 25 years. All of the things you mentioned were already solved adequately in the Linux 0.9 days, and the BSD’s still use old fashioned shell based inits, and work fine in my experience.

In fact, the only time I’ve had issues with any of this stuff has been in the last ten years, and only on systemd-based distros.

The genius of init systems is that they factor all this stuff out, and just handle starting and stopping daemons.

> All of the things you mentioned were already solved adequately

really?

Let's say (not entirely hypothetically) that you have a daemon. This daemon has a config file, which is large and complex. This daemon's /etc/init.d file builds command-line dynamically, then passes it to start-stop-daemon as usual.

Puzzle level 1)

You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?

Puzzle level 2)

Turns out this daemon used LC_ALL (PATH? HOME?) when parsing config, so it works fine when started or restarted from command line, but fails when started as an part of startup process. How do you debug this?

Puzzle level 3)

One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?

The problem of all the init systems is they could be great, but their development just stopped at some point -- and instead of fixing the real annoyances in a general way, they were just saying "this is all fine". Until systemd came and ate their lunch.

Not sure if joking (since, sysvinit is very old and was in need of replacement) so I'll answer in good faith.

> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?

"oh my process failed; let's add `-x` to the invocation."

> You made a typo in init file, so that daemon does not start. The "start" command succeeds, but there is no output anywhere, and process does not start. How do you debug it in the sysvinit world?

as above.

> One day, you happened to be in the server room, watching server boot (you normally reboot it remotely). You have noticed a red message scroll up the screen. This looks important! Too bad it already scrolled off the screen and out of the scrollback buffer too. How do you see it?

`dmesg` or /var/log/messages (or /var/log/dmesg)

Puzzle level 1) ini file:

you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?

I would not call this "easy" or "solved". Possible, yes. Pleasant, no.

Puzzle level 2) inherited env -- I think you made copy-paste error? You just copied the same paragraph again.

Puzzle level 3)

Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.

The sysvinit recommended way is to call things like "log_failure_msg" and "log_warning_msg", and those only print to stdout, which is the console. That's it. Gentoo's OpenRC does claim to have boot console logger, but it was broken every time I tried to use it.

Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.

> you mean, "add -x to the invocation, get back a few screenful of text, wade through that looking for "export" lines and copy them to your terminal, find "start-stop-daemon" line, copy the part of the line and remove irrelevant parameters, and paste into your terminal as well?

I mean, yes? Granted I'm not a developer but printing out exactly what's happening line by line with all variables expanded and all logical conditions printed is awesome, it's something we try to reverse engineer into programs with tools like bpftrace and strace.

Conversely an opaque C application which fails has much less tooling in the way of debugging failures. Especially if your package maintainers have stripped symbols.

> Surprise! Neither dmesg or /var/log/messages or /var/log/dmesg usually contain startup problems! Sometimes the nice daemon might log stuff there, but it is no way universal or required.

Actually, that's the opposite. A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.

> Are you running any servers? If so, you might want to connect to the monitor, reboot and see if any service has error messages. Or you can use console scrollback after the server is booted, but you might need to recompile your kernel to increase is size, it is pretty small. Or just install systemd, it has full boot logging fully integrated with primary logging. And even a special command which lists all daemons which failed to start.

Console scrollback is "nice to have", personally I'm very thankful that it does not use memory keeping things there because there is the odd regression when you configure something (like when you tell IPtables to log deny's: that also gets printed to console.)

re----gardless. I think we're talking passed each other. I had no real issue as a sysadmin with a fully fledged bash, because I can debug it. I get that there's issues with that and most people do not like it- I'm not advocating for it, just saying that what you presume to be "impossible" was somehow solved for decades, and quite competently. It may not be pleasant to you but the alternative is not pleasant for me.

The larger issue I have is that "the alternative" is embedded and entangled itself to everything- there can never be anything better produced.

With old init's, it was a small process with well defined functionality so replacing it was easy. (as evidenced by the many init's among unices.).

Anyway, I'm getting ahead of myself.

If you want to see what systemd could have been, I invite you to look at Solaris' SMF. Please ignore the XML-ness of it, it's "very old" in comparison to systemd. :)

Are you perhaps confusing something?

> A "bad" daemon might not print anything to stdout when invoked (which goes to dmesg) but regardless, this whole comment just reeks of someone who hasn't actually dealt with systems before 2014.

The daemons never print to dmesg -- they may print to syslog, but not all do. The only daemon that I know of that prints to dmesg is systemd :)

To be specific, let's do a simple example. Here is an actual sysvinit file, /etc/init.d/hwclock.sh . It contains:

                if /sbin/hwclock --rtc=/dev/$HCTOSYS_DEVICE --hctosys $HWCLOCKPARS $BADYEAR; then
                    #   Announce the local time.
                    verbose_log_action_msg "System Clock set to: `date $UTC`"
                else
                    log_warning_msg "Unable to set System Clock to: `date $UTC`"
                fi

This version has been around since at least 1998 [0]. Old school enough for you?

So here is a simple question: let's say my hwclock chip/driver is unreliable and it did not work on startup for some reason, so that "hwclock" call is failing. My kernel timezone is not set right, the clock is off, and the system does log_warning_msg to warn you about it. How do I find out about this warning?

To make your job simpler, here is how the how this function is defined on my system:

    if log_use_fancy_output; then
        YELLOW=`$TPUT setaf 3`
        NORMAL=`$TPUT op`
        echo " $YELLOW*$NORMAL $@" || true
    else
        echo " * $@" || true
    fi

Note no dmesg here, no syslog, no other loggers. You can say "bootlog", but it has been broken every time I tried to use it.

Now, nothing is impossible! I can hack this script so it writes to a temporary file. I can re-execute it again, and hope it fails again. But this gets old, you know? I can spend all day debugging early boot shell, but I did it for a long time already, so now I just want to type "systemctl show --failed", see all the failed services, fix it quickly, and move on to more interesting stuff.

[0] https://www.mail-archive.com/debian-bugs-dist@lists.debian.o...

log_warning_msg is a function which is wrapping 'echo'.

if you 'echo' it will go to the console.

Your distro can/does log everything that goes to the boot console.

The function 'log_warning_msg' is provided by the sourced bash file: /lib/lsb/init-funtions ; you can find a copy here: http://www.linuxfromscratch.org/lfs/view/7.0/scripts/apds02....

It's just doing echo.

If you add "echo" it will print. I promise.

And anything printed goes to whatever your distro is using to log tty0 console output. (usually /var/log/dmesg or /var/log/messages)

Which distro logs tty0 console output from echo ? Debian does not. Fedora does not. Gentoo used to, but it broke.

Both /var/log/dmesg and /var/log/messages log printk output from inside the kernel. They do not log echo output from userspace.

Are you sure you are an admin?

> Which distro logs tty0 console output from echo ? Debian does not. Fedora does not. Gentoo used to, but it broke.

None, anymore, that's a little bit the point.

Devuan does, so does Debian 7 and RHEL6.

I just tested it on all three in a VM.

yes, I'm sure I'm an admin.

> Again, I’m not saying that systemd is perfect but that most of the complaints tend not to acknowledge the many problems it solved

It solved nothing but the problems made up to justify its existence. And the ones those faux solutions created downstream. Boot ordering problem? Feh. What a gaslight fest. All caused by chasing 1.2s faster boots. What a colossal waste of everyone's time. This is the same cancer that Shuttleworth tried to foist on everyone with that UI abomination. Servers are desktops are laptops are tablets are phones!