Systemd programming, 30 months later
lwn.net
lwn.net
https://www.agwa.name/blog/post/how_to_crash_systemd_in_one_...
Which made me re-evaluate systemd. Fundamentally creating a monolithic tool to take over from many small single purpose tools seems counter to the UNIX philosophy, and makes it harder to hack on my (mostly desktop/laptop) systems.
On some servers I use s6 toolset. Those are nice. OpenRC is nice. It gets the work done, its so small, its easy to do things with it, and it has less problems than systemd - which I experience on the desktop. Example, started using networkd for managing networks, because hey why not, and NetworkManager is shit, I had gotten tired of typing wpa-supplicant and dhcpcd commands manually. But then, systemd apparantly takes control over the networking subsystem not only the network interfaces - it changes sysctl flags and doesnt delete after itself when asked to stop - it leaves any addresses and other configuration changes it had done. What!?
edit: As all things, systemd-networkd has its place, its just not on a developers laptop, maybe it would be great to have in some kind of cloud environment where you have to configure hordes of machines and virtual machines in them, with networking and networking subsystem configured correctly. Then its nice to have a "do-it-all" tool, but then you should also probably swallow the rest of RedHats output, such as Ansible and look at Project Atomic.
On my servers I do what I've done for about 2 decades now: I ditch whatever init crap my distribution came with and write a single sh rc script that mounts stuff, starts whatever services I want started, and starts getty.
My pet peeve about Systemd is the abysmal error logging for services. Usually it will spit out 50-100 lines that offer zero useful information about why something failed. Adding insult to injury, I have noe easy way to add logging of services. If it were a bash script, I could inject "echo $foo > /tmp/service-debug.log" in the script, figure out the problem and fix it. Now I'm stuck with reporting it upstream and likely getting "Unable to reproduce, closing".
I think it's an elegant and necessary complexity to save us from much uglier and greater complexity.
Red Hat would like SystemD to be the init system of record and have - for years now - been subsuming other utilities, daemons, logging, su, etc. in order to make it the obligate choice.
Any technical gains from the integrations seem secondary to the control this gives them over the direction of Linux as a whole.
This is obvious dirty pool.
If they'd gotten where they were on the merits, I think people would be a lot quieter.
Beyond the political stupidity, systemd has made itself a single massive and opaque point of failure, especially since it ate logging.
Honestly, what kind of complex init system do most /servers/ need?
But the issue is actually that systemd is not just an init system it's steadily increasing in scope to incorporate more and more services.
For example, systemd-timesyncd (controlled with timedatectl) is an excellent NTP system, and after it's been stable for a few years I'll use it with public NTP servers. It takes just one command to get up and running and it keeps your clocks synchronized, what's not to like?
And considering systemd's vs OpenBSD's track record I suppose it's also less likely to get you owned or set your time to Jan 1st 1970 because of some bullshit like "who could have predicted that in 2016 networks are still so bad that UDP can deliver packets out of order".
And, btw, why should an init system come bundled with a time client or a time client bundled with an init system? When can I expect systemd-solitaire?
That's entirely up to distribution packagers. 39k bytes and you get a dead-simple working ntp client daemon (no server) which requires literally no setup. If the distro packagers want to split it out or not include it at all, they can do that and the build system is set up to make that easy.
So it really depends on how you mean "bundled", it's good and there's basically no reason not to include it in the package.
I also figure it's a bit of a straw man to hypothesize a bug which hasn't occurred. NTP, as far as I'm aware, is mostly static structures on the wire. If there's a problem with the client, it's most likely your clock will just be wrong; bad as that is.
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/init/ http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/ntpd/
Alas there is no OpenBSD-solitaire yet, but who really needs that if you can play "Hunt the Wumpus" instead:
My CentOS 7 OSD nodes will sometimes not complete booting because it's hung somewhere in the boot process. There is no way to tell systemd to continue, Ctrl + C doesn't do anything and there is no way to get into the system because sshd is still not running.
Booting it in debug mode to try and get more information makes the whole thing work.
Also, when a unit file gets stuck, it goes into this waiting loop, which is a giant pain in the behind... I want to try and fix it, not wait 10+ minutes for the unit files to time out.
Ctrl + C should immediately stop what it is trying to do and dump me into some sort of login shell so that I can then go and fix the issue...
I've also had issues on Ubuntu 16.04 because of third party software that doesn't correctly set up it's Requires/Wants/Before/After's and then systemd will happily hang the whole boot process. Worst thing is that these issues become more and more apparent on systems with more cores/CPU's... so the 44 core, 88 thread systems we have show the issue way more than the 4 thread desktop that developers are using...
Systemd in their infinite wisdom decided that journald can choose when to log and when not to log something. Historically, syslog etc have been a very thin barrier between programs and outputting logs. Journald now decides if and as and when it's going to bother putting logging data to disk.
One of the things they've decided is you don't really need logs when you run the reboot command. In fact, if you really want logs you have to dig in to their documentation to find this: https://freedesktop.org/wiki/Software/systemd/Debugging/#ind...
That tells you to adjust kernel parameters and write a file to disk to explicitly re-enable logging.
I lost a whole bunch of time recently troubleshooting something on a server running systemd that was interfering with the reboot process, because systemd logged sweet FA. I had a pretty good idea what the problem was, but lack of logs made it significantly harder to nail down. The actual problem is one I've run into all sorts of times in the past, and in every non-systemd case, there's always been enough detail in the logs from the init system to nail it precisely from the get go.
How many of those services have dependancies on devices that may or may not be online/attached at any point in time?
The preach of UNIX culture in GNU/Linux sounds to me like emigrants keeping long gone traditions alive from something grandparents used to speak about.
[1] https://interviews.slashdot.org/story/04/10/18/1153211/rob-p...
For me, the UNIX philosophy has not been about the tools, but about replaceability. Systemd has made itself nigh on impossible to replace.
The SysV init system stopped being convenient for some number of people, so they replaced it, and it has been a long and horrible process. It will happen again if it needs to.
> But nobody is thinking about that because "init-scripts sucked".
Well, now they're thinking about it, and talking about it every time they bring up problems with systemd. Those problems and discussions are something that will be brought up when any potential future init system is being designed or advocated for.
Instead of learning from that mistake, systemd just doubles-down on it. systemd will be much harder than SysV to replace because it includes more than SysV stuff. It will happen if it needs to, but why not just make it easy to replace if it needs to? (answer, IMO: insecurities about your product make you want to entrench yourself instead of constantly improve it)
> Well, now they're thinking about it, and talking about it every time they bring up problems with systemd. Those problems and discussions are something that will be brought up when any potential future init system is being designed or advocated for.
Sure, but why not do that now? instead of pushing it off for people in the future to figure out?
Well, I was actually talking about future successors. What systemd did right and wrong at a technical level, as well as at a marketing, design and adoption level, will obviously have been internalized to a greater or lesser degree depending on the aspect being considered and the person doing the consideration.
> systemd will be much harder than SysV to replace because it includes more than SysV stuff.
I don't think so. There are a plethora of other init systems that have either been started in response, and people work out how to get those into these different distributions. The market for init systems is more vibrant than it ever has been, so any eventual successor gets to reap the benefits of everyone's collective experience in this area. Systemd may be overtaking most the large distributions, but let's not pretend the init system in linux, what it interacts with, and what's required to replace some or all of one init system with another aren't far better understood than they were a decade ago.
> why not just make it easy to replace if it needs to?
It may not be easy to replace, but it's far from impossible, and it's getting easier and better understood all the time because of people that don't want to run systemd. That's a good thing.
> Sure, but why not do that now? instead of pushing it off for people in the future to figure out?
Because regardless of how much it seems like people dislike systemd, I'm confident the majority of people either didn't notice, don't care, or like the new features enough to outweigh their dislike of other aspects. I would be happy to consider data that points to an alternate conclusion, but there's a reason why it's taking over, even if there are problematic aspects.
What UNIX philosophy? The biggest UNIX in the world is Linux, which is a monolithic kernel. The UNIX philosophy lives on in apps, but not down that low.
Systemd is not a single binary either. It is many binaries, every one of them does its clearly specified job and together they live in single repository - just like coreutils.
I find these kinds of discussions a bit strange. I don't want it, it's made my system both less stable and less easily to hack around with. So I'm not going to use it. If you like it, go for it!
Good luck trying that with systemd...
The number of binaries isn't relevant. Claiming that systemd follows the unix philosophies[1] because it has "many binaries" suggests you don't understand that philosophy.
The point is limiting complexity. Smaller, self-contained programs are a lot easier to understand and debug than both a single "kitchen sink" app and a collection of tightly coupled programs and libraries.
If systemd followed the unix philosophy, I should be able to use journald in isolation. Or I should be able to use the init tools just as without bringing in the other tools. The point of a modular design is that each module should be as independent as possible so they can be replaced piecemeal and tested/debugged in isolation.
It doesn't matter if you split the project up into different binaries/etc. Busybox, for example, shoves everything into the same binary, but it's a modular design where it's easy to leave out any module and use the coreutils version instead.
[1] http://www.catb.org/esr/writings/taoup/html/ch01s06.html
The same author also made a followup blog with even more strange arguments. Saying that systemd security features doesn't matter more or less because applications should be secure by themselves / do that on their own.
I'd refrain from forming your opinion solely on blogposts such as these.
Did anyone actually think this was a remote exploit that could be triggered via twitter? I certainly was never under that impression.
a) systemd violates "the UNIX philosophy".
b) "the UNIX philosophy" applies to long-running processes.
UNIX daemons have existed for a long time, and they have never been as single-purposed as the shell utilities; so arguably "the UNIX philosophy" doesn't have anything to say about daemons. Secondly, systemd's PID1 has basically only as much functionality in it as makes sense. If they move that functionality to another daemon, it'll be worse.
> even more strange arguments [...] because applications should be secure by themselves
I'll quote him on this: "good programmers recognize the difficulty of writing bug-free software and understand the importance of designing software in a way that minimizes the likelihood of bugs or at least reduces their impact".
Design and architectural choices including the choice of programming language have a high impact on the code quality and reliability (incl. security) of a system. No matter how good a programmer you are, there's just a lot additional mental overhead when using an unsafe language like C, as you have to always be on the watch-out for type-safety related bugs, and for memory safety (not accessing invalid references), memory leaks (and always free()ing), and bounds-checking stuff.
To quote him further: "Go and Rust are compelling, safe languages for writing the type of systems software that has traditionally been written in C. Systemd is dangerous not only because it is introducing hundreds of thousands of lines of complex C code without any regard to longstanding security practices like privilege separation or fail-safe design"
Forget to check array bounds just once, and you might have just created a new security vulnerability. You have to be perfect. Not make a single mistake. Run Valgrind and every code quality other tool, and you still could have missed something subtle. C is an inherently unsafe language, and a poor choice for new projects when alternatives like Rust and Go exist.
The other problem is that one should try to avoid building giant monolithic systems. (I think) I read somewhere once that Linus Torvalds said that in hindsight not adopting a microkernel architecture was the biggest mistake he'd made.
Quoting the author: "Systemd is far more than an init system: it is becoming a secondary operating system kernel, providing a log server, a device manager, a container manager, a login manager, a DHCP client, a DNS resolver, and an NTP client. These services are largely interdependent and provide non-standard interfaces for other applications to use. This makes any one component of systemd hard to replace, which will prevent more secure alternatives from gaining adoption in the future."
A single PID 1 process shouldn't be doing a million things. Splitting up responsibilities into different specialized programs, running in their own processes would be the way to go. IPC overhead is not sufficiently large to justify creating one giant statically linked executable that does a million things. It's not the Unix philosophy either.
> it's not remotely executable, nor has any relation to being triggered via Twitter
Maybe he shouldn't have have used the word "tweet" in the title, but it's clear what he's talking about a few sentences in. And, what do you mean it's not remotely executable? Did you try running it?
> I'd refrain from forming your opinion solely on blogposts such as these.
The author backs up his claims by linking to specific bug-causing lines in GitHub. He makes several solid beautifully explained arguments. It couldn't be more well-written. How many blog posts claiming security holes link to specific lines of code? Your dismissal of his entire blog isn't justified.
Is that really what you got from that? If so you need to do a much better job of understanding what you read.
My interpretation, for an alternative take, was that Lennart acknowledged it was a problem and should be secured, but noted that there were tools that needed read-write access, so it wasn't as simple as just mounting read-only. He did note how you could mitigate the problem using fstab, but I think a lot of people took his statement that you could do so as the official response and that no fix would be forthcoming, when nothing of the sort was said. In fact, that comment, even though separate from the prior one, had its text encapsulated in parenthesis, which I take as an attempt to indicate it's something he forgot to include in his prior response, and just an addition, so not the definitive answer from the developers on the "solution".
Note, the eventual fix to the problem was at a kernel level, not systemd, because systemd didn't design efivarfs, they just use the accepted mechanism for accessing EFI variables on linux, which happens to be a filesystem. Whether you think that's a good choice for an interface or not, or think it was poorly thought out and implemented even if it is a good choice, none of those things have to do with systemd or Lennart, since they are kernel level issues.
Software has bugs. That's just how it is.
The same can be said about the Linux kernel.
I like systemd and it has helped Linux advance. init was a mess and no one still has come with a golden response that shows the world how anything would be better then systemd.
systemd answered a difficult question and hopefully we willsee more similar advancements shortly for Linux.
Just boot the system and hand it off to a service manager that handles unit files. That was a nice idea. Everything else that came after with tight coupling to systemd was not.
I just didn't want to be forced to use it in case history repeated itself. Opting out of systemd is ridiculously difficult due to that GNOME tie-in. It's one of the most important parts of my system.
I still would prefer not to have systemd, but I will admit that it works reasonably well for my purposes. I'm pretty upset about how it all went down, but that's life in the fast lane I guess. Where are the days when everybody ignored Linux? I should move to BSD ;-)
So they took plan B and used logind, which is part of systemd. People are still free to develop shims that would work with other init systems - but so far nobody is willing to do that.
The main problem is that, as the responder to my comment said, there are not enough outside developers getting in there and fixing the problem. I am included in the list of people who could fix it, but won't because I have other things to do.
It doesn't stop me complaining about it, though. I don't think the complaints are counter productive either, if they are aimed in the right place: Red Hat. Blaming Lennart is useless, whether or not it is fair. It's Red Hat's behaviour that needs to change, because if they replaced Lennart tomorrow the new person would have a mandate to do exactly the same thing.
Opting out of systemd is hard because systemd is the only package which actually solves these problems reliably, and is enthusiastically maintained. Logind is the only maintained implementation of ConsoleKit. xinetd works fine, but it operates outside the loop of the system manager, which makes socket activated services harder to monitor and configure and prevents them from being started before xinetd itself.
Say what you will, but systemd is relied upon because if they didn't rely on it, the functionality they need would not be found elsewhere.
I think these are all growing pains. As the system gets more complex, we are starting to realize that we can't afford to maintain two or three wrappers and sets of configs for the device manager process; we can't afford to maintain a handful of different metadata-enhanced logging systems in each daemon; we can't afford to maintain adapters and configs for a bunch of login managers. This is exactly the same kind of pain which made portable UNIX an overnight commercial success. People yearned not to have to maintain a port for every computer they run on. Today they yearn not to have to have to ask five daemons different questions for the same answer.
* https://www.mail-archive.com/supervision@list.skarnet.org/ms...
* http://jdebp.eu./Softwares/nosh/openbsd-binary-packages.html...
SMF or Nosh
Agreed, but I hope that systemd starts focusing on stability and fixing the regressions they've introduced compared to sysvinit. systemd machines are the only instances of Linux since 1995 where I've had intermittent random hangups during startup and sometimes shutdown. I never thought about startup and shutdown before systemd. It just worked, and if there was a problem, it was reproduceable and a real error (usually configuration), not systemd random fault. I don't use CentOS, maybe their systemd integration is actually reliable.
Statement without substance that sounds more like a "feel good" phrase
> that shows the world how anything would be better then systemd.
Not breaking and being very unstable is a start
> systemd answered a difficult question
No, it caused more problems than solved
> more similar advancements
It's not an advancement except in the mind of some
I'm not against an init tool like Systemd. I'm against 2 things: LP and his contributors don't seem to know what stable software means or even care (I went through something like 4 distro versions before Pulseaudio was minimally stable) and the scope creep of systemd
So you're saying Linux didn't advance in the last few years?
> Not breaking and being very unstable is a start
Statement without substance that sounds more like a "feel bad" phrase
> No, it caused more problems than solved
Statement without substance that sounds more like a "feel bad" phrase
More seriously, show us data that refutes OP's statement instead of simply claiming that it's without value. You do the same thing in your assertions, and it really doesn't drive the discourse forward.
I worked with a number of large enterprise customers running Linux at scale, and when the architects opened up to the value of systemd, they universally sang its praises. Out of all these large customers, I've never run into an issue where systemd was the culprit of a significant, P1 style issue. There's a large, silent majority of systemd users who run into no issues at all.
Linux advanced significantly more under sysvinit than systemd, so does this mean sysvinit is superior to systemd?
> Not breaking and being very unstable is a start
When it works, it works fine. When it breaks, it's very hard to debug. There's plenty of articles out there about this. Even the original article which spawns this talks about the same types of problems - "we have to mask it, not disable it, because <xyz>", or "end users will have to be responsible for masking this service", or "when it was just bash scripts, we didn't have to worry about order".
> There's a large, silent majority of systemd users who run into no issues at all.
Can be equally said about sysvinit and upstart. Does that mean they are just as good as systemd?
My two cents: Developers love systemd; its one more thing they can abstract away and mostly not deal with. People who have to troubleshoot problems with systemd don't feel nearly as charitable towards it.
Talk about a statement without substance.
But setting that aside, I think init is guilty of "breaking" and "being unstable". Having existed long enough for others to work around your flaws is damning with faint praise. The hours I have spent combing documentation upon an upgrade of one thing or another (I'm looking at you, Apache) because init broke would be difficult to count. If you manage all your components from within the distribution then it's usually fine, but if you are managing something not included in the repositories... buckle up, it's going to be a bumpy ride.
But the greatest vote for systemd is that every major linux distribution, even ones that value stability very highly (RHEL and Debian) adopted it, sometimes at great cost to the community. That says a great deal about systemd's advantages in a system often fractured by choice.
The substance is in several bug reports and general trouble that are available in google for all to search
But I'm curious at what problems you did have with init
It doesn't help that we are increasingly dependent on packages that most would argue are as opaque as systemd. For example, if my organization wants to adopt single sign on (and why not), I've got to install shibboleth: a massive, hulking beast of a system that I don't really understand but can tell you for sure does not obey Unix's "one tool, one function" philosophy and with an init script that I pieced together from parts. (Tangent: There are days when I miss Microsoft and it's suite of integrated tools, notable active directory).
So I would say that while systemd has plenty of bug reports, init simply substitutes endless form posts on fixing init for bugs. And frankly, after a rocky start on the switch over, I haven't seen many issues with systemd.
So perhaps it's just acknowledging that best practices don't always get followed and that others have a different set of problems. I expect systemd resolved a lot of issues for distribution maintainers that had to walk gingerly around init when planning their upgrades. Perhaps that's the wrong reason to switch, but at least it's a reason I understand.
"There will be better alternatives in the future that are more secure than what we have now. But adopting them will only be possible if systemd has not destroyed the modularity and standards-compliance that make innovation possible" [1]
[1] https://www.agwa.name/blog/post/how_to_crash_systemd_in_one_...
The article was an interesting example of how upstreaming unit file for systemd means more collaboration between distributions, less wasted work.
This kind of defensive response seeks to shift everything from the technical to the political and leaves no room for genuine criticism, improvement and diversity.
Linux was a success way before systemd, you just can't disrespect all the talented people who have got Linux where it as greybeards or motivated.
You say systemd has advanced Linux but don't say how, you say init scripts were a mess but so are systemd symlinks littered all over the place. This is not reasonable.
They could have easily written this same article when nfs first came out and they were working through their default sysvinit scripts.
If anything, the article is a good example of adapting new tech to work around the many odd behaviours of old tech.
On July Martin Pitt already started to discuss some of the difficulties on systemd-devel mailing list: https://lists.freedesktop.org/archives/systemd-devel/2016-Ju...
During the systemd conference they've been working to create a good solution for this.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394
Gotta go faster faster faster...
These user sessions have existed for various years. Now they're ensuring it'll work for graphical sessions. How is this fast?
Do you really want a real, professional engineer to use the strategy "I don't need verified designs, I'll just make it work in practice" when building a bridge or railway?
'mask' will not start the service, not even as a dependency to another service; instead it will rather let the dependent service fail to start.
Masking removes the unit from consideration altogether, so even if a unit depends on it, it won't start. That means that units with a Requires dependency will also fail, as a dependency is missing.
So typically a fairly top-level unit (like postgresql) will declare itself that it's wanted by a top-level target (`WantedBy=multi-user.target`) which means that enabling/disabling will actually turn the service on and off, unless something else depends on it in which case disabling it does mean the explicit dependency to multi-user.target will be missing, but the unit will be pulled in wherever it's actually depended on.
There are some differences in the details, of course. But a "disabled" service can be manually started.
The functionality that creates these deeper tie ins to Gnome, udev and others today prevent other inits from working for instance with Gnome. Now they want to add a kernel bus. This will make it extremely difficult to replace systemd. Can any supporter of open source and Linux really support this?
A init is a critical subsystem and should have limited scope that is clearly defined, be thorougly tested for that scope and released. Continuously developing and adding features to it does not make for a stable experience. Debian Jessie is now on a outdated version of systemd 2.15 while Systemd is at 2.31.
Things like predictable network names are ironically named as now they are not predictable to end users and scripts that work across systems. Things like eth0, eth1, wlan0 abstract the underlying hardware. If your solution cannot even abstract it and retains the arcane bus names and makes it difficult to even read let alone work with the interface name how is it a solution.
Things like binary logging may be important to a small subset of users and they should have an option to enable it. But why should it be imposed on everyone by default? The technical debt of niche cases should go to those who want it.
https://lists.freedesktop.org/archives/systemd-devel/2016-Ju...
I prefer having my dependencies listed in version control, automatically downloaded, having a tool that can tell me why dependency x is pulled in etc etc.
But once in a while something in the long chain of dependencies fails and after days of debugging you start thinking you should just have downloaded the *.jar files and copied them to libs/ manually.
I think people (in general and not only limited to software development) should strive to minimize dependencies. Otherwise things like this can happen: https://news.ycombinator.com/item?id=11340510
Can't you just check the difference between the last working and the first broken one? If necessary just unpack the last working jar/zip/war side-by-side the first broken and diff them.
> I think people (in general and not only limited to software development) should strive to minimize dependencies.
Here we totally agree. When I started at my current place we had a mix of spring, javaee, hibernate and whatnot. The size is currently down by almost 70% although a lot of that was also unused resources.
My goal is just using javaee + a couple of utilities.
The syntax for adding and configuring plugins is a horrible nightmare, as is the plugin API itself.
Their attitude towards public vs private APIs causes a lot of pain during version upgrades of both the core platform and associated plugins.
Basically, it is the worst Java build system in the world, except for all the others.
Safe, stable, slightly old-fashioned (a bit xml-heavy that is), and gets the job done.
(Wait. Lets keep the xml, I don't particularly enjoy it but it keeps us safe from being overrun by hipsters ; )
Why don't you tell me what I got wrong?
Both have significantly less popular alternatives that often have larger flaws which are often ignored by the proponents of alternatives.
Neither are going away any time soon.
On embedded linux I make, I use everything else but systemd.
I still worry that over time systemd will be integrated so deeply into linux that it will be impossible to remove and also leave no chances for alternatives and if its monolithic approach will be wrong then it will be too late.
Or maybe systemd wants to be everything, to make even linux obsolete
I have an ubuntu laptop which requires a reboot every so often for a reason I have yet to figure out.
I am not sure about the linux communities direction after the recent drive to systemd containers and other complex methods of doing things which should be simple.
Is this an apt comparison for systemd, or is it way better/worse (I'm not an expert, so I'll leave the qualitative opinions to others!) than that?
Exactly what I'd need to do to be able to put a #! on the line before is slightly more complex and you won't be able to reason about it yourself except by guesswork but that's alright because I've got it right. You can trust me.
[Unit]
Description=Some description
[Service]
Type=simple
ExecStart=/the/service/binary
[Install]
WantedBy=multi-user.target
This might look simple, because "Hello world" is simple in most languages. But if you really work with them, you will encounter many hidden complexities.
Systemd being responsible for daemonising also removes several corner cases I could screw up on daemon implementation and saved me time.
All in all I had a much easier experience that I did with similar tasks on SysV.
But in practice, the description of all of that is still simple. Have a look at your service descriptions in /usr/lib/systemd/system - there will be a few services with crazy dbus activation and a long list of options, but most of the others are quite simple and the extra options are only to make it work better, not to make it work in general.
And being able to simply set things like NoNewPrivileges, MemoryDenyWriteExecute, and Protect* without going into LSM profiles is a great simplification.
A colleague of mine had the problem that one of his applications was shut down because the user context which this service belonged to, logged out/disconnected (I think it was a system user). So he had to hack something together to ensure that the service is not affected by this.
No thanks.
Allowing any process to linger around by default is just stupid, which is why "Or just add 'KillUserProcesses=no' to /etc/systemd/logind.conf." is as horrible advice as "just add your own user to the root group".
That’s something no other competitor has provided before.
The way to fix this without disabling the feature is by explicitly putting the process in it's own scope. That doesn't take special permissions to do, it's just that it needs to be done explicitly instead of just ignoring SIGHUP.
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394#221
https://lists.freedesktop.org/archives/systemd-devel/2016-Ju...
This being said, upstart has a similarly simple setup. Any time I have to fiddle with sysv init files, I long for either of upstart or systemd.
Turns out you need to figure out After, KillSignal, pidfile, and some options in uwsgi configuration itself. Oh, and "Type=simple" doesn't work, you need forking.
And this is poorly documented, it's hard to find materials en the web to help you and you are reduced to make many trial and errors with systemd itself giving you few clues of what's going wrong.
It's, however, easier that bash scripts, so it's an improvement.
The documentation is not that bad but declaring with configuration what you programmed with a script has some limits. Basically you have to learn another language on the top of the scripting one that you still have to use in other parts of the system.
Nevertheless I found that past that point it's easier to read a short configuration file than a long init script. I didn't run in big problems so far, but that systemctl reenable is wierd. Why can't it figure it out by itself?
Why would you not use type=simple?
That said, two of us have already pointed out in this discussion where other doco exists for the subject at hand, outwith reference manual pages.
You could have also just copy/pasted the examples from the uwsgi documentation.
Not understanding how the daemon forking works is not the same as systemd being complicated.
sysctl has nothing to do with systemd though.
For now... /s
systemd-sysctl exists already. (Okay, but its not a replacement for it)
In systemd, services are described in a simple file with a few keywords, and systemd tries to do all the magic itself, in a way replacing the scripting part and all the possible code paths that could be gotten that way.
So if something in the system's boot or a process' start-up procedure doesn't work now, it's systemd's fault and not yours or whoever wrote the shell scripts in other init systems; and if the simple configuration file is not complex enough to do something you could do easily with a shell script, than it's also systemd's fault for being too simple.
I'm an old-timer and I like shell scripting, but also think systemd makes writing service files easier and has benefits such as built-in support for cgroups, security features (user/group, chroot, ...) and mini-services like basic ntp server. There were some bugs when distributions started to include it by default some years back, but I haven't had any problems lately.
So rather than building the necessary complexity out of simple, understandable instructions which can be presented on-screen in full, it hides its complexity under a very few complex, frequently badly-named instructions which are certain to be poorly documented even for Linux's standards.
Sign me up! I'll be unfirable!
Those old init scripts weren't simple - to do anything useful you had to copy and paste a whole heap of sh boilerplate...
This is my regular problem in systemd arguments. I understand learning init script - and bash as a consequence - can be tiring, but judging a software by how easy or not is it to configure is utterly and fundamentally wrong.
After a certain point, systemd is not easier to configure than any earlier system; nor is it any more stable, or, god forbid, easier to replace a part of it.
Please stop glorifying things that you're only touched the surfaces of.
I don't care if it's easy to replace bits of - the people who produce the distribution I use can worry about that. All I care about is that it has worked really well for me in production and I have found it much easier to use and maintain than older systems.
Having dependencies specified explicitly helps me to understand how the system works much more easily than cryptic directories full of numbered init scripts ever have.
If you are interested in 'what' and not in 'how' or 'why', then declarative nature of Nix configuration will do nicely.
It can be as simple as
services.nfs.server.enable = true;
services.nfs.server.exports = '' /export 192.168.0.0/16(rw,no_root_squash,insecure,no_subtree_check) '';
One thing I never configured before was wireless in Linux. In NixOS it was pretty much a couple of lines in config files, after that it 'just worked'.
Become an expert user.