Troubleshooting Systemd with SystemTap

Systemtap works beautifully under rhel/centos, but it's a pain to make it work on Ubuntu.

This has slown down adoption by a lot.

We could have had dynamic tracing in gnu/Linux a lot earlier if Ubuntu had done a better job at supporting it.

This is no surprise though: canonical has often refused to adopt stuff that have been thoroughly tested and vetted in the rhel world: think of firewalld Vs ufw. Or selinux Vs apparmor. Or cockpit, that has no equivalent in the canonical world afaik.

I can see a practical argument for safety, but the theory is exactly the same here: both techniques inject arbitrary runtime code into the kernel. That code could do bad things, and so it has to be validated. The location of the validation in the architecture is different, but it's a practical difference.

As far as my personal opinion? ?!@$#!@ just pick one, folks. The time for dueling tracing paradigms was like 8 years ago.

I used to run my home automation stuff (Home Assistant, Node-RED, and a custom shim between my power meter interface and MQTT) in a FreeBSD Jail and recently switched to a Debian VM.

Getting everything to start properly with FreeBSD's rc scripts was a minor nightmare. It took a lot of work to accomplish the task of "Run command x, which doesn't daemonize, as user y in directory z and redirect stdout and stderr to blah.log". For quite a while, I was just running things by hand under tmux because I couldn't figure out the exact method to make it work properly.

In comparison, doing the same in systemd was incredibly easy. This type of service is really easy to set up in a systemd unit file.

I'll admit that a lot of the problems with doing this in FreeBSD were likely my own lack of skill and/or understanding. Systemd doesn't require this level of skill/understanding to set up a simple service though.

In terms of benefits of systemd, the best one for me is logging. Having stdout/stderr from these processes go to the journal automatically is quite nice and querying it with journalctl is easy now that I've figured out how to use it.

When developing simple services (like the custom shim mentioned earlier), systemd greatly simplifies things since I don't need to worry about daemonizing, switching users, logging, etc.

Now though, I really like the systemd service approach - it's so much simpler than hacking custom bash scripts, and it's actually really fully-featured too. For example, it supports lots of different service lifecycles, and even has a watchdog included that can restart failed processes.

I'm still less sure of some of the other parts, like journald, but I'm a systemd convert for services at least!

I've had frustrations with systemd, but the few times I've wanted to write an init script I was glad it was trivial instead of cobbling something together for traditional init.

No more crazy init scripts, no more pidfiles, better templating, cgroups integration OOTB, better handling of restarts, isolation integration (private tmp, capabilities, ...), etc. I enjoy what systemd-the-init-process provides. I see it as a standardisation of what everyone did in a slightly different custom script into a known key-value format.

The real wow moment came when discovering socket activation which allowed systemd to take care of zero downtime deployments of web applications - for me that was reason enough to support the project.

Nowadays I'll use a container orchestrator to achieve the same thing, bit it was a good couple of years whilst it lasted.

In the last couple of days, I became a fan. Not for the many reasons which anyone can look up. But for systemd-nspawn, an awesome tool for spawning lightweight containers.

For instance, you can boot an ephemeral (tmpfs) copy of your system with systemd-nspawn -D / -xb. Several more examples are listed here: https://www.freedesktop.org/software/systemd/man/systemd-nsp...

Many issues with systemd seem not to be with systemd-as-init-replacement, but rather with with systemd-as-kitchen-sink. It's the tight coupling that annoys many people.

Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?)

Compare the upvotes on the .service answers to the alternatives.

Also earlier today I wanted to find out about systemd timers and the first post was this, which thinks they're way better to debug than cron: https://mjanja.ch/2015/06/replacing-cron-jobs-with-systemd-t... - mainly because they have an options to 'run this job right now, in exactly the same environment' - that's an excellent feature, and something cron should have had ages ago.

Getting an impression of systemd from HN is like getting an impression of politics from Twitter. You'll be exposed to a lot of views, but not a wide variety of them.

Then there is systemd-networking. So much easier, simpler to configure than netplan.io and network manager. Again, lots of information presented through journalctl and I never have to deal with networking issues.

I have systemd-analyze which gives me a clear picture of boot times and security issues. From my perspective, these kinds of things are a step above what existed before.

Don't like init style shell scripts? Replace it with something like systemd's service descriptors. It would be fine with me. Now, using this as merge god knows how many different services into one badly written piece of crap that uses 100-1000x more energy than the services it replaces is not an argument that you can win.

I was a relatively early adopter to CentOS 7, and I have yet to run into an issue that I could pin on systemd across any of the Linux systems I use or maintain. Then again, my requirements tend to be pretty "normal" and not terribly taxing - at most I crank out a couple of server-specific service and timer files and get on with my life.

I totally understand where some of the critics of systemd are coming from. No matter how you slice it, more lines of code is invariably going to result in more bugs. And the development team does seem a bit myopic at times - though nothing deserving the death threats they get.

However, there were some clear disadvantages to the old-school sysv init, and I thought that it was important that Linux settle on a new standard init system and do it without yet another instance of fragmentation happening, like with rpm vs deb or Flatpak vs Snap or countless other examples. Systemd happened to be it, but honestly I didn't even care what the init system was as long as something finally "won" and didn't introduce yet another schism. In fact, I highly suspect that if something else had won people would have doubtless found an axe to grind with that one too.

(It's possible other "modern" inits could also do this well, and I do continue to believe that systemd in particular is not the best code even if I agree with its architecture, but it's the best out there as far as I know of the ones that actually exist. Many years ago I ran into an unsolvable bug with Upstart and a particular complex setup that I couldn't even minimize into a useful bug report. I've had problems with systemd, same as I've had problems with, say, Linux and glibc, but they've all been quite solvable.)

The problem comes when you discover that the Systemd version of a service is missing one crucial feature you need or is buggy, so you need to run an external service instead and it starts fighting you tooth and nail. Or when something breaks and tracking down the issue is hindered by it being hidden in behind a bunch of layers of indirection.

For example, try configuring an Ubuntu 16 box to randomize its MAC address before searching for WiFi access points.

That said, it should be eminently possible to have something like service files, and not systemd.

And I must admit, reading the back and forth on the linked issue, I still have zero confidence in systemd upstream as maintainers of systemd.

That, more than the disturbing complexity, "eat-the-world"-tendency and tight coupling (with lip service to separation, by having tightly coupled parts of the overall monolith be "separate" as individual binaries) worries me most about systemd.

https://github.com/systemd/systemd/issues/13674

I no longer have to learn a three-dimensional grid of init systems (init system, distro, distro release).

If this isn't a positive experience...

Systemd requires a small initial study effort in order to be understood.

If you're unwilling to do that kind of things (is: reading the fine manual), you might as well go buy a macbook.

In the end I gave up. I can't put the blame on one part specifically but it was not a good experience.

Other than that I don't see much difference on defining a service with upstart or with systemd, for my basic use.

I don't like the feature creep though.. we have now systemd, systemd-udevd, systemd-journald, systemd-networkd, systemd-timesyncd, systemd-resolved, systemd-homed and who knows what else. worse some of these are kind of irreplaceable in the systemd ecosystem.

systemtap is really like heart surgery on kernel internals that are not stable. So it is expected that it breaks from time to time.

Systemtap is developed by RH people (unless something has changed). So in RH affiliated distros it should just work. But as said, my experience is that they are willing to help with other distros, too.

Or is there a reported breakage in Ubuntu these days that nobody is reacting on?

https://sourceware.org/elfutils/Debuginfod.html