Hacker News Reader

Comment by nine_k | Hacker News Reader

My beef with systemd is not its reinventing things. That part may actually be good.

One problem is that a number of reinventions were poorly made.

E.g. the log format. Yes, unstructured logs have a ton of drawbacks. Can we take some ridiculously well-tested, reliable embedded database or serialization format (like sqlite) and use it as log storage? Alas.

Another problem is the "I know better" attitude.

Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior.

No wonder I don't run systemd on my desktops / laptops.

View on HN

You don't want to use sqlite for logs -- it is way too complex, and in particular, it may roll back (=lose log entries) if it is low on space, and permanently damage data when writing onto disks with errors. It also makes wrong performance tradeoffs for logging -- your choices are either sync() all the time, or roll back on unexpected termination.

I don't like the that journald goes backwards in space and updates indices -- I like "append-only" attribute -- but at least it writes data sequentially one after another, which makes it pretty tolerant of data loss.

As for "without escape hatch", I don't remember any functionality like that? In fact, systemd is full of escape hatches which are documented and supported. Just recently, I had to modify "is computer online" check -- it was amazingly easy, all I had to do was to drop-in a fragment file for the right service. I remember doing the same in upstart -- it was a total pain, requiring reading the source code spread over dozens of undocumented files.

Sqlite is battle-tested though, it has one of the most comprehensive test suite and is used on billions of devices already.

What systemd is doing with this log storage system is reinventing sqlite, because systemd has a fairly new and not battle-tested database implementation, it has shortcomings.

I would argue that either they use plain text logs like before or use sqlite but they should not reinvent a new db.

> Sqlite is battle-tested though, it has one of the most comprehensive test suite and is used on billions of devices already.

That's great, but you've completely ignored theamk's argument that sqlite is the wrong tool for the job.

I don't agree with that argument, sqlite is totally the right tool for the job, it provides consistency, reliability, an api you can query anything upon, transactions, indexing, tools... And fits into files.

You currently can't trust systemd log storage at all, even on my own laptop it manages to have some corruption... Querying is very poor and inefficient, indexing is unreliable... And that's exactly what I expect, building a database is hard.

SQLite doesn't work reliably on NFS, which rules it out for any PXE-booted system with an NFS root.

The issue with NFS and SQLite is that posix advisory locks do not work, or do not work well, on many NFS installations. As long as advisory locks work over NFS, or as long as there is only one client trying to access the database at a time, so that locks are not really needed, SQLite works fine over NFS. NFS is a lot slower, but that's just the nature of NFS and SQLite can't do anything about that.

You can ask SQLite to use dot-file locking instead of posix advisory locking. Or you can ask it to simply ignore locking all together. In both cases, SQLite will work on even a broken NFS system, though with the corresponding concurrency limitations.

Fair. In my experience, the fact that NFS is both pervasive and often broken, for something like a system-wide log manager you can't always assume that your root fs is reliable.

Is dot-file locking a compile- or run-time option?

sqlite is a dirty hack. just because people like hacks doesn't mean it's ready for the serious stuff. it is not. https://research.checkpoint.com/2019/select-code_execution-f...

It's certainly more ready than a custom hacked-up database which has data corruption even on my own laptop.

No local logging will work without disk space, and if the disk is bad anything can get corrupted.

If you are low on disk space you want something predictable to happen: the oldest logs are deleted and the newest logs are written. This would be a nightmare to achieve with SQLite, as said above it's just way too complicated.

If you actually care about your logs you should log to a centralized service anyway, and if any of your computers actually ran out of disk space you should fire your sysadmins/sres for incompetence.

This comment denotes, strongly, the dissonance that occurs when talking about something so general.

It’s my opinion (as an infrastructure type) that centralised binary logging is an absolute necessity above 20 instances.

But that leaves out every person with a laptop, or those people with a very low volume of little instances.

Structured logging is difficult, that’s why logstash is so flexible about input and conversions. And I think that was the impetus for systemd’s journals; to log structurally.

However. The parent is wrong if he thinks decisions should be made under low disk conditions. Having a rotate rule to keep your sliding window of logs to a certain size is fine. But your servers (and laptops, and desktops) should not be taking arbitrary decisions based on this kind of condition.

> This comment denotes, strongly, the dissonance that occurs when talking about something so general.

> It’s my opinion (as an infrastructure type) that centralised binary logging is an absolute necessity above 20 instances.

> But that leaves out every person with a laptop, or those people with a very low volume of little instances.

Yep, it also leaves out people on embedded hardware, and on devices that are in remote locations without access to the Internet, or in a car, or on metered Internet, etc. It's hard to hold in your head just how broadly Linux is used these days.

Does logrotate or plain old write() actually handle those error cases by default?

It sounds like you're arguing to use an even more sophisticated logging system than the status quota of blindly writing bytes to disk.

It's astounding to me the issues people have with this. systemd's existence doesn't preclude anyone from using the old, pre-systemd ways of doing things. There are thousands of linux distributions, and afaik there are at least a dozen which are primarily defined as never-using-systemd.

Face it: the linux userspace has always sucked. Linus punted on it way-back-when (to be fair, he was a solo dev working on a kernel, he had his hands full) and opted for the gnu tools, but those (other than the compiler and libc and a few other spots like grep) have never been best-in-class. It was always just sort of "good enough" to avoid swimming upstream (no pun intended) and trying to swap them out.

Someone decided to make an attempt at making the situation better in one way, doing things differently. Perhaps it was a mediocre attempt, even. (I'm just speaking for sake of discussion here; I haven't read the systemd code, just making a point.). Still, nobody was forced to use it, and indeed many options for avoiding it exist.

Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.

I'm all for experimentation in this space. The old way of doing things was super janky, and I think systemd is an improvement in a lot of ways. I (thanks to a lot more time stuck in my house) have two recent Gentoo installs, one with systemd and one without. They both work, but one works better.

"This is the way we've always done it" seems in opposition to the hacker ethos, to me. Doing things differently to see if there might be a better way seems like it's always valuable, considering that nobody's deleting /etc/init.d/* from your system if you don't participate.

Traditions are not always inherently valuable. Try new things. Discard the ones that don't work for you.

> Traditions are not always inherently valuable. Try new things. Discard the ones that don't work for you.

But that's the trouble. The people who make systemd keep integrating it with everything else they make and vice versa, which makes it harder to try new things because instead of things coming in pieces you might replace, it comes as one big interdependent blob.

That's their business, and their decision if they're the ones doing the work. You can always simply not use that big interdependent blob, same as ever. I'm not sure what the problem is.

I recently installed a Gentoo system with ConsoleKit and KDE Plasma, and it works great, not a trace of systemd anywhere. I also understand the original Gentoo guy forked Gentoo and made a new distribution (Funtoo) where systemd isn't a configurable option even if you want it - it just doesn't exist in that distribution at all, as far as I know. In normal Gentoo you can use OpenRC (default) or systemd.

AFAIK Gentoo also supports Gnome with just logind, too. Haven't tried out that config.

Let me try this a different way.

People say that if you don't like systemd then don't use it -- and that's what the people who don't like systemd want to do. But not all of them want to compile everything from source like Gentoo.

So use Devuan, you say, or some other esoteric distribution with six thousand users instead of a million. Which is presumably what they are doing.

But they still want Debian/Ubuntu to use something other than systemd by default, because you need a lot of people to use something in order to get good bug reports and get people interested in fixing those bugs and making improvements to it. Use in popular distributions is the only way for the alternative to get those things, which makes people who don't want to use systemd unhappy when the most popular distributions do.

One big interdependent blob? Sounds like the Linux kernel to me.

I mostly agree with you. I see two sad things:

* Systemd was not exactly great when it had to be considered "the" replaecment for SysV init (which does suck), and happened to be much more than an init system;

* The big distros like CentOS and Debian (and then many smaller) had little choice but to adopt systemd, because Gnome had it as a hard dependency.

It's unfortunate that of all SysV init replacements, the evolution chose systemd. By its spirit, it's very anti-Linux: it's often seen as "monolithic" even though it technically isn't, because it tries to control nearly everything, and it does not mind breaking the old ways when it sees fit (and not when the old ways are broken).

Quite likely, we're stuck with it as a mainstream solution for another decade, or two, much as we're stuck with X. (And in two decades, Linux the kernel may become irrelevant, due to hardware and OS research progress.)

> Face it: the linux userspace has always sucked. Linus punted on it way-back-when (to be fair, he was a solo dev working on a kernel, he had his hands full) and opted for the gnu tools, but those (other than the compiler and libc and a few other spots like grep) have never been best-in-class.

Which userspace are you claiming is better then? Linux used to offer the best userland experience in the world, because you had control over every part of it, and since things were loosely coupled you could mix and match at will (e.g. prefer BSD grep or Solaris tar? No problem, install them).

> Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.

They didn't build a better alternative and let users choose to adopt it organically. They made a wilfully incompatible, tightly coupled thing, and then pressured other projects to hard-depend on it (e.g. Gnome).

This is quite literally the antithesis of everything that Linux and Unix stands for. The whole reason people were able to use the GNU tools with the Linux kernel is because they are loosely coupled and communicate through standardised interfaces. Systemd by contrast doesn't run anywhere but linux, and you need closely matched versions of everything: your linux has to match your udev has to match your systemd has to match your dbus has to match your gnome has to match your...

If systemd had been around when GNU and Linux were starting out, the whole project would have been literally impossible. It's a betrayal of everything those projects stand for.

> pressured other projects to hard-depend on it (e.g. Gnome).

GNOME wasn't pressured in depending on it. There deciding to depend on logind over ConsoleKit. Initially logind could be used without systemd, that changed after a systemd version due to cgroups v2. There's quite a bit of overlap between the people maintaining ConsoleKit and the people wanting to make use of logind, so there was a push by some GNOME maintainers to use logind.

The arguments against were often quite weak. When pointed out that ConsoleKit was unmaintained for years the answer could often be summarized as "why does it need development".

> It's a betrayal of everything those projects stand for.

Stuff like this makes maintainers of things just ignore you. There might be a valid point in your argumentation, but why bother.

Linux types I've noticed tend to be more holy war orientated. Where did the classic vim vs emacs holy war come from also?

It's tribalism and jealousy, and becomes loudest when one side perceives they have "lost" and don't have the skills to do anything about it.

And that was one of the sillier holy wars, since text editors have near-zero network effects and it's trivial for distros to ship both.

(FWIW I use nano.)

> And that was one of the sillier holy wars, since text editors have near-zero network effects and it's trivial for distros to ship both.

Text editors like vi may have near-zero network effects, but operating systems like emacs and vim have plenty of network effects. If I spend months building an amazing interface to git in Emacs but my colleague uses vim, our organisation is hurt.

That's (I think) why people care: they want to collaborate.

> (FWIW I use nano.)

boggle

I’m mostly kidding about nano, I only use that when I have to. My main editor is HomeSite+ 5.5 running on Windows XP in a virtual machine on my Mac. And I’m not kidding.

Oh and by the way it starts faster and uses less RAM than Eclipse. That's fucked up.

You can say that about Apple users too, but the zealotry has somewhat declined as Apple products has become more mainstream.

Historically Linux users where people that actively choose to use Linux based on a combination of technical & philosophical reasons, thus they are more passionate about it.

Most common Windows users get Windows preinstalled from the store where they bought the computer. It is not really an active choice based on reason, therefore no reason to have a holy war.

The more Linux becomes mainstream, the holy wars will become more & more fringe. Systemd is pushing Linux more into the mainstream.

> Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior.

All those examples you gave are possible to disable in systemd, and in the case of DNS and user process after logout, they are not even the default.

: Killing of user process are enabled by default in upstream but disabled by default in every distro that I know. And systemd-resolved, even if it is build by default, is not enabled by default even in upstream.

Good thing sanity prevails in distro maintainers! But not in upstream, as you duly note.

I also remember that even 2-3 years ago the situation was not as nice yet.

I think what became systemd could have been great software, if not for the, mmm, cavalier attitude of its creators (not necessarily personal; it might be amended by RH corporate deadlines).

"Not in upstream" doesn't seem to correspond with the comment you replied to.

Quoting: "Killing of user process are enabled by default in upstream but disabled by default in every distro that I know."

I really shouldn't write HN comments right after waking up.

The fact that `KillUserProcess` is default to on is not something that systemd forces to your distro, there is even a flag to disable it at build time. It does makes sense in many situations (like IoT, embedded, etc.), that is why it is on, but distro mantainers also needs to know what is better for its users (that anyway is a really small part of Linux users, embedded is much bigger).

resolved has been enabled in Ubuntu for years and in the next Fedora.

This is a decision of your distro mantainers, they could also default to unbound, dnsmasq, bind, or anything else. They use systemd-resolved because it probably solves some problem for them, even if this problem is reduced burden of maintenance.

I had to build something on top of systemd-journald recently for work. I was prepared for the worst after hearing all of the gripes people have with systemd.

And you know what? It’s fucking awesome. Structured logging is just baked in. I don’t have to parse fields back out that have been smashed into a string. I get full timestamps, machine IDs, pid, uid, gid, and all the rest. I get a cursor that I can use to durably resume consumption of logs between restarts. Shit, the cursor even lets me ensure that I’ve consumed logs even when they’re being aggregated from multiple machines, without a single change to my code.

Sure there are a few warts here and there. Everything has them. But it took less than a week for me to be convinced that it’s past time for plaintext logs to die.

This, exactly this - I remember one of the most bandied about reasons for the binary log format was that it made replication over the network easier, or something.

Fast forward 10 years and I still cannot find a reliable way to do this with journald, apart from forcing it to output to syslogd and then shipping the syslog logs.

Linux desktop security was (and largely still is) in shambles. Logind at least tried to force some sanity.

Was is executed well? No, not at all. In typical FOSS fashion it just changed from one release to the other, and that's it.

Yes, there were probably notes about it somewhere in a long forgotten changelog deeplinked 4-5 clicks below the surface of Debian's and Ubuntu's homepage.

Should have they introduced a deprecation period? I don't know, maybe, probably. (But who are "they"? systemd maintainers? Debian/Ubuntu?)

> Another problem is the "I know better" attitude.

All good software is based on this attitude. Wait, I hear you interject. You have examples to disprove this. Well whatever it is you might think isn't the "I know better" attitude is probably just the same "I know better" attitude with one layer of abstraction.

Reply on news.ycombinator.com