Hacker News Reader

Comment by andrewstuart | Hacker News Reader

Before systemd, a difficult bug was a difficult bug.

With systemd, a difficult bug leads to an attack on systemd itself, and questioning whether it should even exist.

I like systemd, it's a pity some vocal people aren't enthusiastic about it.

Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?

I guess my question would be, is this really a general issue about systemd being difficult to debug, or some specific component subsystem of systemd. systemd is a big place.

View on HN

Just like politics and the news, when you read of something being attacked, I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?

Why would someone speak-ill (attack) something they didn't have a bias against and were neutral on?

The question is not does a bias exist, it's if a bias is illegitimate. Many people who have issues with systemd have them because of their own experience, which they document. And often the very act of documenting their issues brands them as biased. When people say systemd is a regression and list their reasons, they are labeled as not having enough experience with systemd, and if they'd only give it a try, they'd see it's better. But that's the exact thing that's occurring, someone tried to do something that used to be obtainable, and their experience with systemd has been things being difficult, obscured, badly documented, undiscoverable.

If we're going to merely ask if attackers have a bias and use that as a reason to downplay or ignore their contrarian arguments, we should be asking if proponents have a bias and also use that to downplay and ignore their supportive arguments. "Are systemd developers in favor of systemd because Red Hat signs their paychecks?" is just as much an invalid assertion of bias, assuming the developers have made meaningful supportive cases that are independent of the bank account that funds them.

Before systemd the error were shell errors.

before systemd the bundled logger wrote to a file, didn't direct people to journald, that lags like a champ when a log file is > 1 meg.

before systemd errors in config files were on specific lines

I don't have bias against systemd I have bias against it's flaws.

I like its config file. I hate it's documentation.

I like that it has a cron like feature, I hate that its pretty impossible to debug.

_whats my bias_? I have >>5k machines to look after, I don't want to think about systemd. (and no, not all of them drain into splunk. that costs money.)

Worst of all, I hate the attitude: every problem is a deep and personal critism.

it's not, in the same way that when a bug brings down your entire estate at 4am isn't a personal attack on me. It's just terribly annoying, and I want it fixed.

I'll take the rare systemd bug over the buggy hell that most sysv init scripts were.

In fact, I run both distros with and without systemd and broken init scripts are a constant headache, while I've yet to encounter a single issue caused by systemd.

Your and zaarn's comments are the classic fallacious dichotomy between van Smoorenburg init+rc and systemd, called out by the Uselessd Guy years ago. One day, the world will have a systemd discussion without an instance of this fallacy cropping up.

* http://uselessd.darknedgy.net/ProSystemdAntiSystemd/

sysvinit has it's warts too and it can get quite ugly.

For example, atm I need to have 3 services. Two of these services need to run at the same time, they can run separately but then they are not useful. The third services can only run with both services online and is first run 15 minutes after boot (not earlier!) and then every hour but the hourly run MUST NOT run before the first run of the boot (ie the very first run may happen 15 minutes after boot, even if the hourly run would have happened in that time, it cannot)

This is not trivially achievable with cron and classical init other than resorting to complicated shell scripts in both.

With SystemD I can cut down on the complexity by declaring this dependency in the service files and the timers in systemd are sufficiently configurable to achieve what I need.

Sure, systemd isn't flawless, but it's a mile better than the average init script I'd have to hack together otherwise. Journald isn't pretty either but with the right configuration you can cut down on the lag.

And before systemd problems like the above wouldn't have errors in specific lines either, some shell script would be subtly wrong and the entire charade would collapse in an instant.

You don't have to think about systemd at all. Ansible and other modern tooling can abstract managing the machines for you (atm I managing about 200 VMs in Ansible playbooks and it works great)

I probably would have resorted to another process manager if I were on sysvinit such as supervisord, god, monit etc to avoid making those scripts.

I used supervisord for anything complex, which is where I think systemd get's it's syntax from.

Anisble abstracts some stuff (ensure running, running as use, etc) but for more complex stuff (process type, dependency management etc) you're on your own shipping system files.

My main beef is actually three things:

1) crap error messages,

2) crap validataion

3) journalctl being a crap VIM wrapper.

of all of them, journalctl is the most obnoxious. Instead of deffering to the system viewer it enforces their crap viewer. It truncates by default, and without restarting the viewer, or forcing it through less. (Why didn't they use less by default? less code to maintain..)

Journalctl uses less as a pager by default, you can set the env variable SYSTEMD_PAGER to specifiy what pager to use (or just PAGER) and SYSTEMD_LESS to specify parameters to pass to less.

journalctl has no own pager implementation and if no pager is installed then none is used.

I give you validation and error messages, systemd is a bit sparse on that front. Would be easier if there was a way for programs and services to signal back error messages to report.

edit:

You can actually enforce non-wrapping by setting SYSTEMD_LESS to FRXMK (the S option that is otherwise included causes truncating, if omitted you get wrapping)

This can all be configured in /etc/profile, .bashrc and friends

Systemd is the light that people don't want to look at and embrace, heh?

What I see is a an appalling amount of arrogance that comes with the "I'll solve all your current and future problems".

Systemd actually doesn't and in fact can't do what it pretends. The evidence is in the never ending list of posts like these that pop up all the time. And the arrogance of pretending to be the top of the init.d hierarchy of init scripts is not sustained by the competence of actually fixing the problems that people keep reporting.

There is a reason for the existence of the systemd opposition, and is similar to the opposition you find in any field where the top of the hierarchy is abusively taken by force instead of by competence. This in fact will be the reason for systemd downfall and I'm confident to say that it will take with it everything that sustained it in it's position.

> The evidence is in the never ending list of posts like these that pop up all the time.

So posts hating on something are proof that something doesn't work?

The open-source way used to be "come up with an alternative and win on merit". These days, since what systemd is trying to do is complex, it seems to be "complain about it online but keep using it" which leads to a super toxic user base, which might explain why bugfixing it (since you have to interact with said users) is so hard.

My understanding is that you are conveniently blind to several facts and this allows you to create a naive narative about what open source is supposed to be and what sustemd is leaving to the rest of us to do. Thus your downvotes.

With systemd, any really tricky low-level bug is systemd's fault, because systemd tries to do everything, in the most "advanced" way. And due gnome3 dependencies it's been adopted by all major distros. Before systemd, a difficult bug was not always the fault of one newfangled opinionated and unavoidable package.

I read the first blog post about systemd when it was written, and thought it was really interesting. Then I read the second blog post and thought "eh this is getting a bit complicated". And since then of course systemd has relentlessly taken over and added much more ... so of course practically any low-level system bug, randomly distributed, has a really high chance of being in systemd. And again, it does lots of stuff that was never needed before - in this case it's un-sharing a mount namespace for extra isolation of a small trusted service, which is overkill and complicated, and this struggle to figure out what the hell happened seems particularly unnecessary.

>And due gnome3 dependencies it's been adopted by all major distros.

That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files. That is very clear from nearly all mailing list discussions.

Also, people can and do use gnome3 without systemd. (https://wiki.gentoo.org/wiki/GNOME/GNOME_Without_systemd).

> That's not correct. It has been adopted by all major distros due to ease of use and maintaibility of unit files.

Or rather, because unit files work the same across all systemd-using distros, they can actually be upstreamed to the application developers and distros just package them, which is a huge time-saver compared to having to maintain init scripts for hundreds of daemons.

Makes sense.

timesyncd is a network connected daemon. Why would you not isolate it as much as possible?

The job of synchronizing the system clock is not a job that belongs to the init system. It is a job that belongs to people who can write and debug the very complex cases of NTP.

The init system simply has to fire it up, take it down, and report on whether it's still running. Any further elaborations are features of the init system.

All the problems mentioned here were failures of the init system in doing things that the init system should be doing, but were hard to diagnose because this particular init system does not and never has put "easy to debug" as a high priority item.

"as much as possible" would be to put it in a virtual machine, or actually separate physical hardware, maybe in a multi-million-dollar bank vault to prevent physical tampering.

Then there's a reasonable amount of isolation, which would be to run it under its own user, which can only write to one or two directories, and can't read user directories or non-world-readable system config files. This is already common practice. (Creating this user at package install time, or in the initial base distro config, is completely sufficient. And simpler.)

"as much as possible" is really not worth it (and sucks resources away from implementing reasonable security practices elsewhere).

It's disappointing to see the top comment here condoning ad hominem attacks.

But if you do believe that the identity of a speaker is more important than the arguments they make, note that the author of this blog post has a history of praising systemd (e.g. https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdRight) There is no anti-systemd bias motivating this post.

>>condoning ad hominem attacks

I'm not condoning ad-hominem attacks. In fact I am pointing out that often it is systemd itself that it the victim of ad-hominem attacks, which is to say that often people attack systemd, not its behaviors.

But you wrote:

> I think "who is saying this, doing they have an underlying bias against this thing they are attacking"?

There is nothing wrong with examining the motive behind someone's argument. That's very different than an ad hominem attack.

That is literally an ad hominem attack: https://en.wikipedia.org/wiki/Ad_hominem

later parts of the page claim the ad-homs are legitimate. Most people would consider a claim of "ad-home" as implying illegitimacy though.

In this case it is literally the definition of the fallacy. The argument was not addressed at all. Not a single word of the post went to even acknowledge the content of the article.

Whenever I see a post that does nothing but dismiss an argument based on who made it, I do question the poster. There is a chance it is warranted, but in those cases evidence is at least presented. This post didn't even offer that.

As kemitche says, you have incorrectly said that I attacked the blog post author.

There are so many systemd haters out there that I immediately wonder about the motives when I read a new attack on systemd. Pointing this out is not attacking the author.

"I'm just asking the question" is hardly a valid defense against ad-hominem. If you're wondering about it and don't take the effort to research then that's your problem, not something you should share with the world. If on the other hand you have done the research, but can't be bothered to share the results the question is again why share the question with the world without sharing the answer?

Systemd is not a hominem - it's a piece of software. It doesn't have feelings.

It's understandable that people don't like critique, especially delivered harshly as it so often is on The internet. But that only applies to those who worked on systemd. What I don't understand is why it seems to have fanboys.

Because every opinion can be used to build groups and people like to belong to groups, so you have a systemd fan and hate group.

> I like systemd, it's a pity some vocal people aren't enthusiastic about it.

To give some context as to why, for non-C++ developers systemd makes things very difficult to debug and understand in general.

Moreover the biggest issue /I/ personally have is the fact it was essentially forced on me by the mainstream distro maintainers.

When you discuss the merits and fallbacks of systemd it’s common for people to either try to discuss its merits only against sysvinit. Or they imply that issues that have affected me personally are not such a big deal and I should be quiet because: progress.

It’s a frustrating and exhausting position to be in. It’s like everyone shoving Windows registry on Linux down your throat, with all the opaque, obtuse and stifling qualities that would come with such a solution and telling you and your ilk to be quiet and suck it up.

Edit: I’m getting pounded by downvotes. But I guess offering any kind of other opinion about systemd for any reason isn’t met with kindness. However I can say this: neither google nor amazon use systemd to host AWS or GCP. At least consider why that is before you bury your head in the sand.

Systemd is straight C, not C++.

spins finger in the air the difference is immaterial and irrelevant in this context.

A huge part of GNU/Linux is written in C. A small part is written in C++. I would accept the argument that C++ is less accessible than C to people maintaining Linux systems.

Based on this alone I would say that yes, the difference is material and relevant.

pish posh. It could be written in Egyptian and it would make no material difference to the ease of debugging for most people.

The number of people who are C++ programmers and can deal with strace vs. C programmers who can deal with strace is really that significant? I don't buy it. The two groups intersect significantly.

The point is that other people who are not C or C++ programmers have to deal with this kind of stuff from systemd.

This is just pedantry for the sake of it.

Systemd would be only marginally harder to debug if it was in C++ or heck, even rust, or go.

No one is opening the source code here.

> C, not C++

The two wretched sisters.

> Uses UNIX-like OS

> C/C++ bad

> writes C/C++ every day.

> C is is stupid and breaks things

> C++ has serious mental health issues

> Moreover the biggest issue /I/ personally have is the fact it was essentially forced on me by the mainstream distro maintainers.

So pretty much the same as any other decision distro maintainers make, if you disagree with it then.

Most things can be replaced. Not systemd.

Prefer postfix or exim over sendmail? Sure, completely replaceable. Prefer another cron system, Logging system. Prefer another /anything/ and it used to be possible to swap.

Not any longer.

Distro maintainers didn't force you to do anything. They made a decision about how to allocate their limited resources. And we're past the days of Linux being a primarily volunteer OS, but a fair proportion of distro maintainers are still volunteers. They are not at your beck and call. If you want to have a Linux distro without systemd, there's several. If none of them meet your needs, pitch in with time or money to improve them. The purpose of free-as-in-libre software is that you can do the work yourself, not that you can expect others to do the work for you.

False dichotomy. I pay redhat a lot of money, short of paying people full time to maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything.

Some decisions cannot be unmade. Short of starting from scratch with gentoo (which is a Herculean effort at scale) I’m not sure what you’re saying. I cannot be disgruntled at choices because distro maintainers time is worth more than mine? Fair. But how many people have to be in my shoes for it not to be fair.

The company I work for runs Gentoo at a global scale...

Then you either make significantly more money than we do or are able to allocate resources to that project as it is key to your business strategy. The main focus of my company is to make Video games. Not maintain operating system distributions. That’s why we pay redhat so much.

I think Gentoo scales very easy. Especially in enterprises.

One thing I like about Gentoo that no other mainstream distro has is something like this https://packages.gentoo.org/packages/net-proxy/haproxy.

That page says that you can install any version of haproxy that suits you. You don't have to upgrade/downgrade the OS just because you need an older or newer piece of software.

But, as always, no distro is perfect. Gentoo has its issues too.

You live in a better place!

> maintain a mirror distro of redhat without systemd, which would be impossible given how embedded it has become into everything

Not RH-based, but have you checked out Devuan (systemd-free Debian)? I've always valued RHEL's (up until 6) stability, but I'm seeing Debian/Ubuntu becoming more and more used in enterprise roles where before these had already been strong in web roles. I'm also seeing a need for a systemd-free minimal O/S for container (not container host) roles.

> False dichotomy. I pay redhat a lot of money

Well why are you paying for a distro which doesn't meet your wishes then? Why not choose another one?

The author has gone into a lot of details on the issue so your question about whether 'this is a general issue' is answered quite clearly in the article and the follow up itself, including a link to Poettering's rationale and write up for dynamic users.

Characterizing a bug report with a detailed write up and follow up solution as 'attacks' and 'bias' is inexplicable.

Yeah, it's more like "Any time someone presents a problem with systemd, someone else will claim politics."

It's kind of fascinating to watch software become so polarizing. Like, it's technology. It doesn't inherently have feelings or bias. But we make it so.

You obviously didn't read the article before deciding this wasn't systemd's fault.

The bug is the lack of any error handling in the DynamicUser feature of systemd. The original failure is not logged anywhere - and launching timesyncd just goes ahead even though the setup for it fails completely. Naturally timesyncd then fails to start.

(Or perhaps the bug is the whole DynamicUser feature. From reading the bug report, it looks like it will fail any time anyone has a user-only FUSE mount point on their system...)

The other part necessary to trigger the bug is that uid 0 is not root and is not able to read the user-only mounts. root would still be able to read the user-only FUSE mount.

User 0 is root, and it is not able to read everything on other machines — as has been the case for decades.

Perhaps I misunderstood this line then?

> The machine that this happened on is an NFS client and (as is usual) its UID 0 is mapped to an unprivileged UID on our fileservers. On this machine there were some FUSE mounts in the home directories of users who have their $HOME not world readable (our default $HOME permissions are owner-only, to avoid accidents). When systemd was setting up the 'slightly modified mount name-space' it attempted to access these FUSE mounts as part of binding them into the namespace, but it failed because UID 0 had no permissions to look inside user home directories.

https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdDynami...

My reading is that (a) systemd can't read home directories on the file server, because clients understandably don't get root permissions there; and (b) the reason systemd was trying to look in those directories is that they contained (non-root-users') FUSE mount points.

Reply on news.ycombinator.com