Hacker News Reader

Comment by jbnorth | Hacker News Reader

Maybe I'm just out of the loop but what is the controversy regarding systemd?

There are many. Most amount to complaints about sysadmins having to re-learn a lot of stuff, much of which didn't actually need to be changed, at least not so thoroughly. Early on, most of the complaints centered around the poor migration path, the abandonment of plain text log files, and the absurdity of high-level user-facing software like the Gnome desktop environment having a hard dependency on a particular init system that many felt was still kinda half-baked. systemd as a project has also been a poster child for feature creep, and while technically many of the new features that have been added over the years are optional, in practice few distributions make it easy to partially opt-out of new systemd stuff. And through all of the controversy, the systemd maintainers have shown a stunning inability to acquire diplomacy skills; systemd has succeeded more in spite of their ability to promote it, rather than because of it.

Pretty much. There are components of systemd I really like. systemd-logind is really neat, it makes dealing with drm (direct rendering) devices nice by having a service where you can request access to GPUs without bumping into the problem of one process opened the endpoint and has exclusive control. It also makes dealing with machine events like closing your laptop's screen simple if you use a minimalistic window manager like i3.

systemd-resolved is nice if you're using systemd's network configuration system, but it's a pain when you're trying to run your own DNS server.

udev really shouldn't hard depend on systemd. I perfectly understand having some systemd specific integration, but why make it mandatory?

There being said, various people have been working to cut it into separate components (eudev, elogind, etc.)

> udev really shouldn't hard depend on systemd. I perfectly understand having some systemd specific integration, but why make it mandatory?

udev is part of systemd now, because the systemd developers were the only ones willing to maintain it.

Doesn't the existence of eudev disprove that?

(This is a serious question; I'm not trying to start a fight)

I think the existence of eudev proves that somebody is now willing to maintain a udev, but it wasn't always the case.

Eudev was forked in the immediate aftermath because systemd only cares about systemd. Even Torvalds strongly criticized systemd-udev maintainership[1].

1. https://lkml.org/lkml/2012/10/3/484

> at least not so thoroughly.

So you still thought some updates were supposed to happen but it wouldn't have happened if it wasn't for systemd as people are too lazy to touch any mostly working legacy bits.

Starting this off with a disclaimer that these are things I've heard or read, and aren't necessarily a statement of my views.

The controversy mainly surrounds the claim that it is a single package that tries to do way too much by itself.

Calling systemd an init system isn't correct. systemd contains many things, one of which is an init system.

The other claim is that the systemd developers aren't open to much outside opinion, but have power over the community because important parts of basically all desktop Linux systems are now part of systemd and have strong dependencies on it (such as udev).

For example, systemd timers (timed systemctl services) are like cron if cron was slightly more complicated and had very good logging. :3

journalctl is a text adventure fever dream. Do systemd timers use a different logging system?

You can output journald to syslog with one config option[1].

1. https://www.freedesktop.org/software/systemd/man/systemd-jou...

Then it should be on by default. Having to use journalctl instead of reading a plaintext file in /var/log should have resulted in a swift slap upside the noggin the moment it was first considered.

I otherwise don't care about systemd, except for the more-than-one locations I might find an init script hiding on my system. It's fine.

But don't make me use journalctl just to tail a god damned file like a normal person.

Edit: I may have reached that point on friday night where the bottle of port and my grey beard have finally reached "old man yells at cloud" stage. My apologies.

> Having to use journalctl instead of reading a plaintext file in /var/log should have resulted in a swift slap upside the noggin the moment it was first considered.

I absolutely love the idea of journalctl (without commenting on the execution).

As an admin, there was always the chance that $DAEMON_OF_THE_DAY put their logfile in a particularly creative place, thus sending me on some wild goose hunt. Now it's just `journalctl -u $DAEMON` and that works immediately, every single time.

Also, as a user running journald on a notebook, I love that it's just one config switch to send all the logs from all the services to /run instead of /var, to reduce the write load on the SSD. Sure you could fiddle with a bind-mount but that would have to happen very early in the boot process so I don't think it would be as reliable.

As a daemon writer, it's one less option for me to care about. Especially when my daemon is just some bash script, logging to stdout is the easiest way.

And logging to stdout also meshes well with running the same daemon/script in container environments like Docker or Kubernetes where containers usually don't even have syslogd. (In fact, I have containerized a service that requires syslogd and that was kind of a pain. I ended up using a mini-syslogd implementation that just dumps all logs on stdout so Docker can pick it up.)

It also allows you to read the logs in JSON, which is fantastic for automation. Parsing the syslog as a text file means a single newline in a syslog message screws your effort up. With journalctl, you can sift through error messages from specific services, confident that the results you are getting are exactly what the logs contain.

Unless, of course, some of the binaries are corrupt.

What is the advantage to having to write all the log messages twice instead of once? What is the disadvantage in having to type "journalctl -f" instead of "tail -f"?

My usual "tail" is not tail -f, but tail -n 100. In fact, I jump around a ton doing tails and heads with varying linecounts, and if it gets unweildy even after all of that, that's when I pipe to less and look at it.

My major justification is I use tail and head constantly, every single day, and it's rarely on system logging. It's java logs, it's python logs, it's rust logs. I'm not going to be giving up tail or head any time soon.

Now I have to type "journalctl --help" every single time I want to look at a system or daemon log, which isn't super common, and when I do need to I'm already frustrated something isn't working.

Put another way, this is the exact WORST time to throw up more roadblocks over something. That's why I assert it should be logged twice, and for those who don't want it to be chewing up diskspace or want to relocate it, well, that's where you can configure it.

I just want my old log files back in their old locations without forcing me to sift through man pages when all I really want to do is figure out why the fucking bluetooth keeps dropping connection. I understand why others see this as a great step forward but it sounds more like the windows registry and event viewer to me than it does the configuration file and log file approach I adore from my unix systems. I'm sure the registry and event viewer has a lot of devoted fans as well; the difference is it's been there for most of my life (or at least, the parts where I was doing more than playing X-Wing vs Tie Fighter), which is why I mostly just wandered away. I'm not a sysadmin, I don't try to be and I don't want to be, and it seems like systemd is hellbent on forcing me to be one whether I want to be or not and I am so not here for it.

Then it's "journalctl -n 100". Or "journalctl | less".

> Now I have to type "journalctl --help" every single time I want to look at a system or daemon log, which isn't super common, and when I do need to I'm already frustrated something isn't working.

I don't think that keeping users from ever having to learn anything new is a good justification for doing things a particular way.

> I'm not a sysadmin, I don't try to be and I don't want to be, and it seems like systemd is hellbent on forcing me to be one

Personally I don't understand this sentiment. As a user I have found that systemd has made using Linux systems much simpler for me. Yes, I had to read a manual, but I don't think that is anything like "forcing me to be a sysadmin". Didn't you have to read the manual for tail at some point in the past?

Nope, tail is such a trivially easy program to use. There's like 2 switches that matter to me, one of which requires a value as well. Keeping things super simple is a really important thing, and I guess I had taken it for granted that everyone using gnu-style tools would be on the same wavelength - for the most part, at any rate. FWIW, I can't bear to read the awk or sed man pages either.

I fundamentally agree with you; learning something new is not necessarily a bad thing. Forcing you to learn something when you're actively trying to figure something utterly unrelated out is where it gets dicey. Not everyone wants to do the "Malcom in the Middle Hal Changing a Lightbulb" routine when there's already a Very Important Thing to Fix.

The problem with systemd is that it took something that already worked perfectly fine for my needs and said "let's overengineer the everloving pants out of this! get in the car grandpa, it's time to roll!" I didn't ask for it, I have WAY more important (to me!) things to do, and it's actively running interference with that. Surely you can see that level of frustration, right?

Put another way, I'm sure it's a LOVELY bikeshed for some. Some people have wanted a bikeshed this fantastic and wonderful for a long time, and they will use this bikeshed with great joy daily.

Meanwhile, every time I'm trying to get something I actually care about done, it's like pulling teeth. Maybe I'm using it wrong, and I'll be the first to admit I have the patience of a toddler when it comes to reading the man pages. What I can DEFINITELY say, though, is that if systemd never existed my quality of life would be doubtless better than it does with it existing. That's really the only metric I need.

Like I said: definitely in old man yells at cloud territory, maybe crossed with a dash of lazy man commits to further laziness and myopic man can't see the forest for the trees. I'm not saying you are bad to like systemd, I'm saying that it made changes I didn't ask for, it didn't do it in a way that was painless and approachable without a time commitment, and I've yet to be satisfactorily convinced that the value proposition is worth it - not for my uses, at any rate.

What I hate is not that it logs to a journal log, it is that the binary db used by journald is non-standard and undocumented.

This means that a dead system can potentially not be read by a live system, as the journal can have undocumented breaking changes.

It is documented, but the documentation is unfortunately not authoritative: https://www.freedesktop.org/wiki/Software/systemd/journal-fi...

Also, they couldn't make breaking changes to the format or it would prevent people from reading their own old log files after upgrades

Fuck, not this shit again.

If I get some 3rd party documentation, that says it may or may not be correct, and the implementation is literally the only authorative documentation then it isn't actually documented.

> they couldn't make breaking changes to the format or it would prevent people from reading their own old log files after upgrades

This is the point. They do not guarantee that you can.

Nope, they go straight to journald.

It tries to do too much, badly, and with an air of entitlement around intentionally breaking things.

Change is good, so let's change away from systemd.

If someone is able to create something that is half as good as systemd then distros will switch time it.

The thing is that systemd solves a lot of problems for distros in a good enough way (frankly, currently in a way that is better than any existing alternative) that they are happy to have systemd deal with those issues while they focus on the stuff the distro makers are interested in.

What is desireable to a distro maintainer is offten not what is desireable to a user or admin.

Systemd favors the distro maintainer and cloud application service provider, and maybe the appliance manufacturer.

It's the exact opposite of a useful tool empowering the the individual.

As an admin and desktop user I like systemd. I have to write less config and less boilerplate code to shore up dependencies across networking, mounts, and things like DNS than I did back in the sysv days. That time saving enables me to get on with other, more useful things.

As a lowly JS developer, I have had a positive experience with systemd. One service file and it handles logging and process management. Better than the insane options the JS community cooked up IMO.

> If someone is able to create something that is half as good as systemd then distros will switch time it.

Not a given, since one of the issues with systemd is its integrations. Replacing init now requires you to replace your system logging and session manager, at a minimum, as well as replace all your service files.

systemd-logind is optional and you would be able to use ConsoleKit instead if anybody had maintained it in the last 5 years.

Unlike e.g. upstart in its native mode, systemd can run SysV init scripts.

https://unix.stackexchange.com/questions/233468/how-does-sys...

By "system logging" yjftsjthsd-h was referring to systemd-journald, not systemd-logind.

True, but on that point xe was correct because it's the one part of systemd that is neither PID1 nor optional; I was assuming session manager referred to systemd-logind.

The latest round of controversy is probably because They're pushing the new 'homed' component, which is supposed to change up how home directories (and user accounts) are managed.

It's supposed to enable easier encryption of home directories, but breaks ssh logins.

Therefore it's probably only suitable for multi-user systems (which need to protect home directoried from other users) that are also never accessed remotely - a subset consisting of approximately 0 systems.

Actually AFAIK the main usecase is single user client systems (say a laptop). Homed makes it easier to encrypt the home directory at any time (without the need to reformat with LUKS) and handles stuff like suspend better (can drop encryption keys before going to suspend and then requesting them on resume).

that's such a small sliver of linux users that the vast majority of linux users (who actually need SSH) are taking the overhead for nothing in return...

Don't use it if you don't need it. The developers made it very clear that systemd-homed is only meant for single-user laptops.

Home directory encryption is a terrible security mode for single user laptops (an attacker with disk access can simply modify the unprotected root file system and exfiltrate data after the fact). Whole disk encryption (and encrypted suspend to disk for bonus points) is simpler, more secure, and doesn’t break ssh (and all sorts of other things, like batch jobs).

Mostly that it WAY more complicated than it needs to be, and it often replaces well written and tested tools with VERY poor implementations.

Things like NTP, syslog, DHCPd, cron, iptables, mount, automount, handling /tmp, /dev, DNS, su/sudo, etc.

Letting the init system talk to the network before userspace has the ability to setup firewalls gives me the heebie jeebies.

Often the implementations of systemd don't handle failures well, leave off critical features like encryption, DNSSEC, etc. They are often hard to debug, and generally worse than the features they replace.

Not that Linux didn't need a nice init system, but that systemd is trying to take over everything. The lastest volley on that front is taking over management of /home with homed, but in a way that's incompatible with ssh.

Do you really want to depend on systemd's homed for managing /home? I'd rather not.

Interesting that you list DNS, because in my experience one of the worst-implemented network protocols in pre-systemd Linux is the DNS client. The systemd implementation isn't perfect either, but it's a lot better.

The design of classic Linux DNS simply assumes that 100% of Linux systems are university lab machines, and that there is a single DNS server that is 100% reliable.

I've heard people making various arguments to support the classic Linux DNS client's failings as if they were features. These have included: that I should "fix" my telco's DNS servers (wtf!?), set up a HA load-balancing solution on premises (in my home!?), use anycast routing (really!?), reinvent the DNS protocol to not require multiple servers (I'll get right on that), or change third-party software to not require DNS (wat?).

When pressed, all of these people eventually boiled their arguments down to: Fine, yes, it's broken, but it's the way it is, that's what the man page says, that's what was established in the 1980s, and systemd is different so we don't like it!

It's mind boggling how conservative Linux people are...

I had zero problems with Linux’s dns client until about 3 years ago. The systemd one and whatever it replaced took 5 seconds to resolve anything remotely last time I checked. (It did cache after that, fwiw).

This was on some Ubuntu LTS. 18.04, maybe? I paved the machine, installed a distro that doesn’t chase the new shiny every six months, and moved on.

Most of these things are simply untrue. The idea that homed is "incompatible with SSH" is preposterous.

> Letting the init system talk to the network before userspace has the ability to setup firewalls gives me the heebie jeebies.

Are these just random claims? The init system is userspace, and you can have any service wait for anything to access the network, including disabling the whole network target and triggering it manually.

Here’s a remote exploit for the systemd DNS client: https://groups.google.com/d/topic/linux.debian.user/2KyNDWMj...

Here’s one for systemd DHCP: https://blog.erratasec.com/2018/10/systemd-is-bad-parsing-an...

Firewalls wouldn’t help with either of these, in fairness.

Okay, how do you get the ssh folder contents to validate the remote user, if validating the user is required to decrypt?

You can't. And the systemd answer is that you simply throw away ssh.

> The idea that homed is "incompatible with SSH" is preposterous.

And yet, any honest assessment of the pros and cons of the homed proposal has to discuss the impact on SSH users in the cons section.

> Letting the init system talk to the network

What?

Part of it is that it is a more or less monolithic package that started as an init system and now does more than some people would like.

This results in certain packages switching to require systemd modules(gnome & udev) and essentially locking systems without systemd out of using said software. Communities supporting non-systemd init systems now have to maintain forks of these modules so that they can keep these packages supported on their distros.

The other issue, which is the issue I care more about is that it does these things in a way that breaks compatibility with existing workflows and exposes it to a number of different issues due to its increased complexity.

Don't get me wrong, systemd is excellent in many ways but like many others, I am not a fan of it overall and choose not to use it where possible.

An example of something very controversal about systemd: journald.

journald is a replacement for the system journaling daemon and is a generally very good journaling daemon. It is in many ways more performant than other journaling daemons and can be easier to use as well. The issues however come largely from the fact that it uses a binary log format. This breaks the ability for other tools access logs nearly as easily unless explicit support for the interaction is added by either journald or the tool in question.

I find systemd journals much more accessible. Instead of passing a log file to the tool, you pass journalctl's output to the tool, and you can serialize the output to any of the supported formats (incl. json) and use much more tools, more directly, than you could previously.

Compare that with random older default debian log line from one of my systems:

    Feb 14 01:15:30 shakes kernel: [    4.014481] [drm] No driver support for vblank timestamp query.

Is that from 2008 or 2020? Noone knows. You have to do a lot to even be able to select by date correctly in your log processing tools. Such a basic thing.

My biggest issues personally are the lock-in of systemd overall and for journald the binary file format. If I could use journald on its own with OpenRC first and foremost and preferably have it store logs as text be it json or whatever then I would use it hands down.

Journald just works for the most part and is much more convenient in most regards nowadays. If I remember correctly, interop was more of an issue before the big distros started to switch to systemd but now things seem to have mostly worked themselves out.

With regard to my comment about wanting logs stored as text, it really just comes down to having had to deal with corrupted binary files one too many times and unless it is absolutely unavoidable I find myself preferring text files for that reason. I'd imagine file system based compression leaves most if any storage benefit from binary logs rendered moot.

I know you can run a secondary syslog daemon from the output of journald or alternatively use a job to backup to a text based format but it would be so much more convenient to be able to just set a config setting. Having easy config and things "just working" is supposed to be one of systemd's strengths anyway.

Final thing to note is that at least from the perspective of someone running Gentoo w/ OpenRC, issues with traditional loggers are typically config mistakes more than anything. As for your debian log example, syslog-ng which I believe Debian had used should default to isodate for the datetime format. At least this is the case after RFC 5424 was adopted in 2009. That should give something along the lines of "1985-04-12T19:20:50.52-04:00". Nowadays unless the defaults are changed, most logging systems shouldn't have that datetime issue.

Hmm, that particular system is up since 2008, so maybe it inherited some config from that time during upgrades. :)

Not really on topic but this brings up a pain point that I wish more distros addressed. Sane defaults are obviously important and over time what classifies as the "ideal" default config changes. Most package managers seem to apply a default config on the first install and then never update it again.

At least from my experience, the only distro I have used that has offered any facilities in the package manager for handling config updates is Gentoo. Whenever there is a package update, if the default configuration files for the package change during an update, you are notified to run `etc-update`. This command then allows you to diff your current config with the new defaults, choose to use the entire file for the current config or the new default, or patch certain lines from the new default into your existing config.

Personally I find this super convenient, especially because whenever you perform a package update or install it will let you know whether you have any outstanding configuration files or important upgrade notes to address.

I'd be greatly interested to know if other package managers have these types of config and service management utilities. I looked around but I didn't really find anything.

> Maybe I'm just out of the loop but what is the controversy regarding systemd?

The problem with Systemd is that every time they introduce new non necessary feature they break something in the system. https://bugs.kde.org/show_bug.cgi?id=417038 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Do they run tests ever?

Now when I launch my desktop I have to login in tty console first, start tmux, detach it, logout from tty, then login into desktop session normally. Otherwise logout from desktop session will hang and logging in again will fail.

They suggest using `systemd-run` instead, but it's broken too at this time. https://github.com/systemd/systemd/issues/3388

The issue with systemd is that it is developed in a similar way to traditional UNIX systems like Solaris, BSD etc.: the low-level user-space parts of the OS are developed as a single project with a coherent design vision, common configuration syntax, documentation, etc.

Naturally this is very controversial and people complain that it is against the UNIX philosophy.

My pet-bitch is that it's far too large and convoluted to audit properly for most-sized teams.

As is the Linux kernel.

Yes, but the Linux kernel’s not replacing a few thousand lines of shell script.

systemd comes from a gnome developer. a gnome developer doesn't care what came from before and doesn't care what they produce. all a gnome developer cares about is the future.

the future is what is all that matters and getting there is what matters. have you heard the phrase "the end justifies the means"?

that's it.

Reply on news.ycombinator.com