Systemd by Example | Hacker News Reader

Systemd by Example

systemd-by-example.com

496 pointsFoxboron69 comments

This is outstanding... I really wish I would have had this learning systemd as it would have saved me _hours_. Let's be real - a lot of Linux fundamental stuff is still pretty terse. Learning tools like this really smooth it out and optimize how long it takes to wrap your head around a concept. Also, I am a _huge_ advocate for hands-on in regards to retention.

Adding to my favorites and will be passing this on over the years - thank you for such a great resource.

If anything the biggest problem the systemd docs have is verbosity. It's all classic UNIX man pages, a billion pages of detail on every possible setting with no useful examples anywhere. Fortunately the core system is simple enough that the learning curve isn't too steep but I'd really hate to try and learn it from the official docs. They don't even apply CSS to the HTML versions of the docs.

Wikipedia: https://en.wikipedia.org/wiki/Systemd

Web: https://systemd.io/

Src: https://github.com/systemd/systemd

Systemd manpage index: https://www.freedesktop.org/software/systemd/man/

https://www.freedesktop.org/software/systemd/man/systemd.htm... :

  man 1 systemd
  man systemd
  man init

...:

  man systemctl
  man journalctl
  
  man systemd.timer
  man systemd-resolved

The Arch Linux wiki docs for systemd are succinct: https://wiki.archlinux.org/title/systemd

Fedora docs > "Understanding and administering systemd" https://docs.fedoraproject.org/en-US/quick-docs/understandin...

RHEL7 System Administrator’s Guide > "Chapter 10. Managing Services with SystemD" https://access.redhat.com/documentation/en-us/red_hat_enterp...

Well, we also have a very nice manpage viewer.

I think systemd.directives(7) is an often overlooked manpage.

https://man.archlinux.org/man/systemd.directives.7

FWIW, the man.vim vim plugin does grep and some syntax highlighting. https://github.com/vim-utils/vim-man

TIL about that manpage. I've always been guessing at which level in the hierarchy would contain a particular option ("Is it service-specific? Or maybe applies to all execs? Or maybe applies to all units?") and just giving up and opening them all in parallel.

Couldn't agree more. Once familiar with the big picture this page is the best starting point whenever to you want change something.

To get the big picture I read the systemd posts on https://0pointer.net/blog/archives.html But that's 10 years already! No idea what would be recommended today.

Yes, the Arch docs are the best. But they aren't from the systemd project itself, are they?

And, as a Debian user who often finds himself on the Arch docs - there's a ton of distro-specific stuff in the Arch wiki. And rightfully so - it's a wiki for a distro.

These sorta landmines when trying to just research/digest a concept can really suck. OP's tool really eloquently breaks things down to _just_ core concepts so you can quickly start to grok what I consider to be a relatively complex tool.

In my experience very little of it is distro-specific, especially if you're on a distro that has similar components as Arch. I'm on OpenSUSE Tumbleweed, which also uses systemd, etc and is a rolling distro with the latest versions of everything, so a lot of the wiki articles apply directly as long as I translate the package names.

Agree - but there's still bits and pieces that don't translate over to other distros which can suck if you're jumping into to learning something. When I read the Arch Wiki as a Debian user I realize I'm reading it through the lens of someone using a different distro.

Overall I use the Arch wiki very often and it's because of the exact point you're making - I'm just being pedantic saying those slight distro differences can be a pain.

This isn't a knock but it's exactly what I was talking about and why I find OP's learning tool to be so valuable.

Lots of folks learn by example and hands-on labs. Personally, I'd much rather learn the basic ropes by jumping into a tool like OP's vs. finding/digging through all of these resources. I'll also criticize to say you likely already know much about systemd, and were able to pull/filter these resources much easier vs someone completely new to the concepts.

To illustrate further: vim is another tool that has outstanding learning resources, everything from very quick "hey get started" examples docs all the way up to adventure games. If I had to go back and relearn vim I would absolutely do it this way vs. digging on man pages like when I was a kid in the 90's. Personally, I learn by doing.

---

Overall - OP's thingy is what I would call a "rich interactive learning tool." It's anecdotal, and obvious projection - but _for me_, interactive learning tools optimize the time it takes to fully "grok" a subject from scratch vs. jumping into a bunch of docs/man pages.

I often find the `## Usage Examples` heading in manages to be most helpful, too.

~Examples as Integration Tests as executable notebooks with output and state assertions may incur less technical debt.

How to manage containers with [MicroShift] k8s/k3d with systemd might be a good example.

Read the Source, Read the Docs, Write the Docs, post the Link to the actual Source of the docs: https://github.com/systemd/systemd/blob/main/docs/index.md

Systemd man pages source: https://github.com/systemd/systemd/tree/main/man

Also https://www.freedesktop.org/software/systemd/man/systemd.mou... https://github.com/systemd/systemd/blob/main/man/systemd-mou... :

  man systemd.mount

And almost none of the tools help you naturally discover the missing feature you didn't know to solve the problem in the most "systemd*" way.

The docs are not friendly enough to introduce you to new tools, and they're mostly shallow enough to not explain the technical details of how things interact or where all the defaults are stored. They exist but clearly this page exists because they need to improve.

I know, I know, pull requests welcome....

At that point I'd call the docs a developer guide rather than the docs a sleep deprived sysadmin needs at 3am to fix a server

Agreed, I have generally avoided systemd as I don't feel it's fully baked yet (not trying to start a flame war, that is just my experience and I'm sure it's fine for others). Still, I need to really learn it both for when I am ready to use it daily and just to have it in my wheelhouse for supporting others. A guide like this seems like a great way to go "hands on" without spinning up an instance and fumbling through it blindly.

Hopefully this guide will stay up to date given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.

>Hopefully this guide will stay up to date given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.

The core dependency principles around targets, services and startup does not see a lot of change, so this concern is not really realistic. You'll see more of this around the supporting utilities (networkd, resolved, and so on).

It's on version 250. How many stable releases do you need to feel that it is "fully baked"? The software is of course evolving and new features are added but breaking changes are rare.

The number doesn't mean anything really, give me the access control and we can have a thousands times more in a week.

The number has a meaning because it is the result of the release process at the end of a development cycle

That will still be true when I take it to 250,000. The development cycle will just be shorter

When you admit to needing to learn something in the same breath as throwing shade it completely undermines the shade you were trying to throw.

That doesn't follow. If anything, I would expect knowing, for practical reasons, you have to learn something you think is dumb would make you even more angry about it, not less... or, maybe, you just believe something being either popular or common means it must be good? :(

That doesn't seem very practical. Viewing something as dumb before learning anything about it is holding a prejudice towards it.

It's reasonable to have the opinion that juggling chainsaws is a dumb idea, even before you've learnt the techniques for making it slightly less dangerous.

Is that really any dumber than doing any similarly dangerous industrial jobs that require a lot of training? Or something else dangerous in entertainment like being a Hollywood stuntman, or being Siegfried & Roy? I'm sure you could get a lot of money for having that skill if you did it in the right place.

If you found yourself locked in a prison, you might suddenly be motivated to learn how to survive there, even if you're not a big fan of imprisonment.

> break things, wontfix

In fairness, quite often I've found this is often a case of exposing existing brokenness rather than creating new problems, but it doesn't make it any more comfortable when it happens.

>given the "move fast, break things, wontfix" approach the systemd authors currently have with the project.

I don't understand where anyone is getting this impression from, that describes the opposite of my experience with systemd. The authors publish stability guarantees, and they have done so for several years: https://systemd.io/PORTABILITY_AND_STABILITY/

I love systemd both for it's actually functionality and the fact that it's almost like the flagship of a whole movement to make Linux more consistent and integrated and less of a pile o scripts.

But for some reason, the authors occasionally say something that feels completely out of place for the project. They fix important stuff eventually, but sometimes it gets confusing to watch them discuss it.

The "Invalid username runs a service as root" bug is a good example of something that they didn't seem to take seriously. But it's fixed now.

Also, PulseAudio. Pulse is good. It's not that great and was deployed before it's ready on some distros. The whole Pulse/Jack split that is only just now being fixed was annoying.

Some people might still judge Poettering based on 10yo bugs in Pulse rollout?

After a lot of recent events, I don't feel comfortable judging any open source project on the speed at which they fix bugs or the way they communicate about it. In open source there is no obligation for anyone to fix anything or communicate anything. To me it's a miracle when anything gets fixed at all. It feels like the community is still largely held together with duct tape and string.

Yeah, I just expect all software to be buggy. To be honest my main way of evaluating software is by Googling stuff like "X ate my data" or "X stopped working" and seeing how many major complaints people have, relatively to how many users there are.

I think the duct tape and string feeling comes from the obsession with smallness and modularity and people trying to make "Linux not be like windows".

Half the FOSS scene WANTS things to be taped together because their main thing is random tinkering.

A guarantee is only as good as the willingness to honor it. There are great ideas and some good implementations in systemd, but when the people behind it have a pattern of refusing to fix (and sometimes even acknowledge) show-stopping bugs, it makes me apprehensive about relying on it.

Again, I didn't want my comment to start a heated discussion about systemd, but inevitably that happens. If I had praised it someone else would have come along to tell me how much it sucks (I don't think it sucks, personally, I just have issues with its current development process).

You can't just leave accusations like that without links to concrete issues. The discourse becomes Kafkaesque.

https://lists.freedesktop.org/archives/systemd-devel/2014-Ma...

https://bugs.freedesktop.org/show_bug.cgi?id=76935#c10

----- Adding "hacks" but not testing to ensure said hacks don't cause issues:

Lennart Poettering 2014-02-21 13:49:25 UTC

To make this work we'd need a patch, as nobody of us tests this.

Comment 3 Michael Shigorin 2014-04-04 06:30:57 UTC

Hope all of you either test all the combinations or do not break at will those you don't have the time and inclination to test, at least in system-wide components that are not specific to systemd, while pushing the latter hard.

Comment 4 Lennart Poettering 2014-04-04 14:56:43 UTC

Well, cgroups-less kernels are explicitly not supported by systemd. However we added some hacks to allow it to boot to a certain degree even if a lot of things will not work correctly afterwards. In this mode when you boot you will actually get a warning on screen and bootup is delayed by 10s to make sure the user understands this.

Now, this mode recently broke, and it will segfault early on. I am happy to take a patch to 'fix' this again, but I will not work on this as i dont run kernels like this, and as mentioned its not really supported anyway...

Another option is to simply be honest amd stop supporting in entirely, and refuse booting completely. And I figure this is what I will eventually do if nobody cares enough to send me a patch to fix that segfault.

https://bugs.freedesktop.org/show_bug.cgi?id=74589

I can't figure out what the problem here is, are you trying to run systemd with a kernel with cgroups compiled out? I don't see how that's a show stopping bug, from the discussion there it was never supported.

No, I'm not. I'm satisfying the other person's request for sources with examples I've come across in the past. In this case it was not about cgroup-less kernels but about Poettering's and Sievers' attitudes towards fixing the issue.

This is why I don't like to talk about systemd on any internet forum; it always leads to people like you ready to attack any position around the subject, and me having to defend a position I don't even hold (I don't think systemd sucks, I think its developers could do better though). I am done with this thread.

I'm not attacking you or asking you to defend a position, I'm asking for clarification because I legitimately do not understand what the issue is. I totally agree with you, it is not helping anyone to become defensive over this. I don't understand what the attitude was, it seems to have been fixed 8 years ago? Did I miss something?

There is a trillion of tutorials for every young and hyped technology out there: Svelte/FastAPI/Elm etc. However try to learn how to maintain and secure a server or a database or anything fundamental... and it can be really hard to find good and updated material.

https://systemd-by-example.com

I just tried the first exercise of creating a default target and this web application is a great learning resource for Systemd. It should be noted that it works best as a complement to the author’s series on Systemd by example¹.

I also love how the web application works without requiring multiple JavaScript dependencies hosted on third-party servers.

¹ https://seb.jambor.dev/posts/systemd-by-example-part-1-minim...

I was just wondering why everything loaded so fast and was so fluid...

I will up-vote every single one of these "_______ by example" sites that are made. I'd also love to see a "Buy a Coffee" or quick 5-10 USD donation button on each one - and if you want a commissioning for one throw a patreon or kickstarter up and I'll share it around.

I found the gobyexample site to be invaluable when I first encountered the language, and now I send people there BEFORE I send them to the official go docs (which is saying something - because the official docs are great!).

Here's hoping we can get well-structured documentation that scaffolds like these to replace most medium posts in the top results of most Google searches! :-)

I was very anti systemd initially but mostly because it meant the many initd scripts I had for different services would no longer be usable or the daemons I had would no longer need to self mange their own pid files. But mostly it was out of fear - knowing how many battle scars each of those shell scripts had learned or forking lines of code had adapted could be lost… fear that maybe systemd would not carry those lessons forward… I let go and it has been pleasantly easier and much less stressing then I imagined… more examples like this are great thanks!

Wow. This is amazing. This is the bar needed for many learning resources.

Thank you.

Since systemd I never had to learn about my init system, it basically just works. For to reasons.

1. On desktop the old init systems were quite good already, before systemd was introduced. Systemd made it better, as I've been told: I never had any beef with it (oh boy did I have beef with init systems in the old days, not only on desktop...), I did not do any benchmarks to see that it shaved off a few seconds in startup time (and helps remove lots of fragile network mgmt code).

2. On server I now use Docker. It has no init system. And when I need one I use one that fits the docker world (i.e. supervisor).

> It has no init system.

Apologies that I can't link directly to the "--init" flag but docker actually does have an init, it's just (err, was?) compiled into the binary: https://docs.docker.com/engine/reference/commandline/run/#op...

My recollection is that it either adopted, or inspired, https://github.com/Yelp/dumb-init#readme which folks used to put into their Dockerfile as the init system back in the day

Folks (ahem, I'm looking at you, eks-anywhere[0]) who bundle systemd into a docker container are gravely misguided, and the ones which do so for the ability to launch sshd alongside the actual container's main process are truly, truly lost

0: https://github.com/aws/eks-anywhere/issues/838#issuecomment-...

I recently switched to multirun [1] which in my case is superior to supervisor. I don't have to create any config files anymore, and it behaves exactly as i want it to: If one on the processes dies the complete container will die and docker's restart-policy takes place.

[1] https://github.com/nicolas-van/multirun

...and no need to keep a python interpreter in memory :)

Good tip, I would like something that that's offered by my $distro.

I like all those 'learnXbyexample' sites, though I really hope I can have a drop down menu or side-bar to access the TOC instead of going back to homepage each time for next subject.

Just the other day I noted on Twitter how bad the official systemd website is for users. I'm looking forward to this site recovering from HN hug.

The missing documentation. Just hope they manage to keep it up to date.

The feature creep here is immense despite the claims it's modular because individual executables do fixed tasks...

Don't get me wrong I understand the power of systems but it's not written with a OG UNIX mindset in mind which is a little sad.

Maybe slightly off topic, but I'd love to hear about people's experiences using systemd as process manager inside a container, specifically for the unprivileged container case, where systemd would be launched in user mode, so only doing process management, without all the other kernel-hook stuff going on.

I really like systemd, and I really don't like supervisor. So I'd love to figure out how to make this work.

I know the easiest thing would be to ignore the user mode business and just use podman, where the work has already been done [1], but at least in the short term that doesn't help for more restricted environments like cloud kubernetes.

[1]: https://www.redhat.com/sysadmin/improved-systemd-podman

May I ask why you'd want to use a process manager inside a container? I just consider Docker my process manager. If my application crashed, Docker will deal with it.

A valid question— certainly there's a school of thought that says one process per container, end of story.

However, there are lots of applications out there (especially legacy ones, think stuff like Zoneminder) which are made up of multiple small daemon-type processes sharing state across ports and maybe even the filesystem, and have a strong reliance on system services like cron or log rotation. Yes, with effort an application like this could be be fully "ported" to a container-native setup, but the path of least resistance is often to just make the container environment present as being more like a full VM.

I get the impression a lot of the entire point of containers in those things is to deal with those awful zoneminder style designs, PHP junk that needs specific Apache configs, etc.

I prefer to just avoid any application that requires a service to in any way be configured to account for the app.

It's bad enough that Zigbee2MQTT is a separate process, but it's acceptable because it is a pure layering model and as long as you are local only and not needing passwords, there's no manual config.

Things like ZM that need an actual database are just insane. Multi-page configuration instructions for something that the(Sadly closed source) AgentDVR does with one process, no containers, no major configuration to conflict with anything else.

But modular apps exist, so we need containers to make them act like monoliths.

I've reverted from podman to docker as the podman ecosystem isn't there - docker-compose, cadvisor, some networking - all require tinkering to run.

My interim solution is running docker containers as user, using a simple USER directive. Straightforward and still added security benefits.

Systemd can be used in a docker container only when lot of system services are masked. See my old CentOS 6 with systemd in docker container project as example: https://github.com/vlisivka/docker-centos7-systemd-unpriv

That's fascinating— thanks for pointing it out. I definitely assumed the issue was with the daemon itself, and not with the services it was trying to start.

https://developers.redhat.com/blog/2016/09/13/running-system...

This article led me to believe that I could run systemd in an unprivileged docker (not podman) container, but I did not find that to be the case. It was not happy until I added `--cap-add=SYS_ADMIN` capability, and then it started working.

Centos docker images have a copypastable example in their docker hub description.

I just learned the absolute basics of systemd the other day while trying to set up an auto-restarting game server on an RPi for my little sibling, and finding helpful examples was extremely difficult. This seems like a perfect solution!

Out of curiosity, how is this implemented? Is there a javascript VM running in the browser, a remote VM or can systemd be run in a container?

There is a link to a post at the top which gives a detailed overview https://seb.jambor.dev/posts/systemd-by-example-the-playgrou...

but the short of it is it creates a limited remote container and pipes the CLI back and forth.

Thanks!

Wow wonderful set of articles. A big thanks! Wanted to understand this stuff for a long time but could not give enough time to navigate the dense documentation.

Site appears to be down

HN hug!

Reply on news.ycombinator.com