Systemd can be a cause of restrictions on daemons
utcc.utoronto.ca
utcc.utoronto.ca
This can help troubleshoot issues and makes experimenting with systemd options faster.
I think there's been some talk about adding a built-in way for systemd-run to copy settings out of a .service file, but it doesn't exist yet.
I've written Perl/Python scripts to do this for me. They're not really aimed at working with arbitrary services, but it should be possible to adapt to different scenarios.
https://gist.github.com/dextercd/59a7e5e25b125d3506c78caa3dd...
There are some gotchas I ran into. For example, with RuntimeDirectory: systemd deletes the directory once the process exits, even if there's still another process running with the same RuntimeDirectory value set.
Both in terms of artificially reducing the number of CPUs you expose, but also in terms of enforcing a memory limit that will kill all processes in the build before the broader kernel OOM killer will act, in case you screw up the number of CPUs.
But the one thing that I always wonder is about (virtualization?) in the sense of something like docker just for containerizing or some sort of way of running them in some sort of sandbox without much performance issues or something, I am kinda interested in knowing what might be the best way of doing so (is podman the right way or some other way like bubblewrap?)
Edit: just discovered in the comment below the (parents parents?)comment that there is systemd isolation too, that sounds very interesting and the first time I personally heard of it hmm
Bubblewrap really is more for sandboxing "transient" containers and being able to separate specific things from the host (such as libraries), with other applications handling the image management, which makes sense because its primary user is Flatpak and Steam. Once the application inside the container is exited, the sandbox is destroyed, it's job is done.
Podman is a Docker clone, it's for development or persistent containers. It will monitor containers, restart them, can pull image updates, setup networks between them etc.
They both use namespacing and cgroups under the hood, but for different results and purposes.
Your right that systemd has sandboxing too, and it also uses the same features as the kernel. Podman can also export it's services to be managed by systemd.
There's literally so much choice when it comes to making containers on Linux.
> but podman handles things like networking, resource and image management
> the sandbox is destroyed, it's job is done.
> Podman can also export it's services to be managed by systemd.
> There's literally so much choice when it comes to making containers on Linux.
Because some stuff is shipped and supported as a docker image and it's just easier to run it via podman which supports nearly all the same options.
I'm unsure if someone has made a tool to convert docker images to systemd. If not, that'd be a pretty handy one.
given that podman can also have a (nicer?) transition to docker is a plus as well.
There are a lot of paas nowadays which use docker under the hood. I think I would love seeing a future where a paas actually manages it using systemd.
I think this might be really nice giving an almost standard way of installing software.
I really want to try to create something like dokku or some gui for making systemd management easier but I will see some alternatives currently, thanks for sharing it!
The best thing about systemd is also the worst thing: it's monolithic. You can containarize applications lightly all the way to having a full fledged VM. You can run as user or root. You can limit system access like CPU, RAM, network, and even the physical hardware. You even have homed which gives you more control over your user environments. There's systemd mounts[1], boot, machines, timers, networks, and more. It's overwhelming.
I think two commands everyone should know if dealing with systemd services is:
- `systemctl edit foo.service` to create an override file which sits on top of the existing service file (so your changes don't disappear when you upgrade)
- `systemd-analyze security foo.service` which will give you a short description of the security options and a score specifying your exposure level.
I've replaced all my cron jobs with systemd jobs now and while it is a bit more work up front (just copy paste templates...) there are huge benefits to be had. Way more flexibility in scheduling and you're not limited to such restrictions as your computer being off[3]
[0] https://news.ycombinator.com/item?id=45318649
[1] I've found mounts really helpful and can really speed up boot times. You can make your drives mount in the background and after any service you want. You can also set timeouts so that they will power down and automount as needed. That can save you a good amount on electricity if you got a storage service. This might also be a good time to remind people that you likely want to add `noatime` to your mount options (even if you use fstab)[2].
[2] https://opensource.com/article/20/6/linux-noatime
[3] You can have it run the service on the next boot (or whenever) if it was supposed to run when the machine was powered off.
Another issue I just ran into is that a colon separated value for ExecSearchPath doesn't work in systemd-run/-p. You have to specify each path as a separate -p argument.
There are some minor annoyances like that, but it's not too hard to work around.
Seeing that podman can run containers as systemd services (see https://codesmash.dev/why-i-ditched-docker-for-podman-and-yo... ), it seems like using containers other than as a distribution mechanism has few advantages, and many disadvantages in terms of dependency updates requiring container rebuilds.
I see it as _exactly_ the opposite. Podman gives me more or less the same security controls as systemd and the package/delivery problem is solved.
Call me when `systemctl pull ...` fetches the binary and everything else needed to run it _and_ puts the .service file in the right spot.
importctl pull-tar https://example.com/image.tar.gz && portablectl attach image
https://docs.podman.io/en/latest/markdown/podman-systemd.uni...
That would be pretty awesome, actually.
That seems like a `machinectl` task though.
That was my point, basically.
You have two options:
1) the usual `curl` or `wget` to fetch the binary and the lib(s) and all the work of validating and putting them in place and the like and _then_ you can use a systemd/.service file to set up controls for the bin
2) podman pull and then either ask podman to make a .service file for you or write your own
because only one of the two approaches has solved the package/distribution issue, containers are _not_ "less relevant given that systemd can twiddle the same isolation bits"
One of the advantages of a real package manager is that it checks signatures on the content that is downloaded. The supply chain on a linux distro's package repos is much harder to break into than typosquatting into a docker registry somewhere.
Every single thing has a sha hash so verifying that I actually downloaded what I meant to download is easy. This gets tedious if I have to `curl https://github.com/someUser/someProject/release/latest.tar.g...` and also get the `tar.gz.sha256` file (if they even publish one ...).
Curl supports resuming a partial file (assuming the sending server also does) but it can't "know" ahead of time that the first 1/3 of the file I am downloading has already been fetched because it's also used by $someOtherArtifact.
> One of the advantages of a real package manager is that it checks signatures on the content that is downloaded.
So does docker/podman.
> The supply chain on a linux distro's package repos is much harder to break into than typosquatting into a docker registry somewhere.
Perhaps. For every "secure" package repo, I'll show you a much more up-to-date package in AUR/Nix.
Back in the old days (late 90's, early 2k's) as a sysadmin I'd often write scripts to chroot or in other ways isolate services rather than run them as root, so extending the init system to handle those features feels like it's a logical extension, not a incompatible replacement.
For software management in R/W environments, there's the podman + systemd combo that'll let you run containers like normal systemd services.
I dont get the distribution "advantage" building em with mkosi but I'd argue it a weakness as far too many are running containers with who-knows-what inside them.
Do you work in the software industry?
Containers are as relevant as ever, of course.
Podman?
There should be an strace-like tool that would collect a log of such "access denied" erros for troubleshooting. Even better, each service should run in its own process group, and tracing could be switched on / off for a particular process group.
It's normally recommended to attempt the access and handle the denial, instead of doing two separate steps (checking for access and doing the access); the later can lead to security issues (https://en.wikipedia.org/wiki/TOCTOU).
Some references:
- https://docs.arbitrary.ch/security/systemd.html
- https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db...
Systemd is strictly better than what came before it, while Wayland still has missing functionality and breaks a lot of use cases.
i am willing to bet that there are systemd haters out there that love wayland and would make the exact reverse claim.
Wayland's weakest point is a11y and automation tools, which XWayland doesn't work for.
> sure it's not the same, but unlike systemd vs sysv-init, you can't run two windowing systems side by side with equal privileges unless maybe you have two monitors and graphic cards. one has to be the one that controls the screen. and the other must necessarily run as a client inside it. wayland-on-X may have been possible, but it would have limited waylands capabilities and development.
You can do both, actually; XWayland can run an X server in a window, and many Wayland compositors will run in a window on top of an X11 server. It's not seamless, of course, but it does work.
It's not "gaslighting" it's just name-calling and argument through insinuation about other people's characters, rather than substance. It's not even ad hominem, because you assume people are arguing in bad faith because of the positions they've taken, not because you know a thing about them.
Whereas with Wayland the effort to transition is significant and most compositors still have limitations X doesn't (and yes, I realise some of those means X is more vulnerable) - especially for people with non-standard setups. Like me.
I use my own wm. I could start with ~40-50 lines of code and expand to add functionality. That made it viable. I was productively using my own wm within a few days, including to develop my wm.
With Wayland, even if I start with an existing compositor, the barrier is far larger, and everything changes. I'm not going to do that. Instead I'll stick with X. The day an app I actually depend on stops supporting X, I'll just wrap those apps in a Wayland compositor running under X.
And so I won't be writing blog posts about how much I hate Wayland, and hence the quotes around "hater" above. But maybe I will one day write some about how to avoid running Wayland system-wide.
If Wayland gave me something I cared about, I'd take the pain and switch. It doesn't. Systemd did, so even if I hadn't liked it better than SysVinit, I'd still have just accepted the switch.
If I one day give up Xorg, my expectation is that it'll be a passive-aggressive move to a custom franken-server that is enough-of-X to run modern X apps coupled to enough-of-Wayland to run the few non-X-apps I might care about directly (I suspect the first/only ones that will matter to me that might eventually drop X will be browsers), just because I'd get some of the more ardent Wayland proponents worked up.
Wayland has a few years to go yet and I'm sure it will be worth the wait. For me, it seems to work OK already.
Nope. No simple way to run a single simple script on boot, which lived in /etc/rc.local for decades. Create a unit instead. It's not much but systemd is full of little annoyances like this one.
> [] offering useful additional capabilities without taking away the old.
Once they introduce capabilities, the old quickly becomes irrelevant due to sheer mass of systemd crowd.
After they have introduced timers, it suddenly became clear that cron, which served us perfectly well for decades is now cubersome and seriously lacking. The same can be said about syslog/journald drama. Now they decided they want sudo replaced. Up next: something else that you cannot even think of, is already on their radar.
Have you tried?
At least on Debian and Ubuntu, if /etc/rc.local exists and is executable, systemd-rc-local-generator will auto-generate an rc-local.service unit that will run rc.local.
If the unit isn't visible for you, create an /etc/rc.local file. chmod +x it. Then run "sudo systemctl enable rc-local.service". Then try "systemctl status rc-local.service".
If your distro has the relevant generator, you'll see something with a line saying "Drop-In: [some path]". If so, "systemctl start rc-local" should run your /etc/rc.local, and/or it will run after reboot.
In other words, you can easily have /etc/rc.local running on boot.
On Debian and Ubuntu, you can do ls /usr/lib/systemd/system/rc* to see the compatibility options for SysV init that are available to you.
If your distro doesn't ship with an rc.local drop-in and generator, then put this in /etc/systemd/system/rc-local.service:
```
[Unit]
Description=/etc/rc.local compatibility
ConditionFileIsExecutable=/etc/rc.local
After=network.target
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
RemainAfterExit=yes
GuessMainPID=no
StandardOutput=journal+console
StandardError=journal+console
Then run "systemctl enable rc-local.service". Now, as long as /etc/rc.local is executable, it will be run on startup after your network has come up (you can change the "After" if you want to trigger it based on another condition instead)
> clear that cron, which served us perfectly well for decades is now cubersome and seriously lacking.
Cron is severely limited compared to systemd timers, but nothing stops you from also running a cron. I have anacron running on my systemd setup because I agree it serves a lot of needs perfectly well. It coexists just fine and lets me use a cron for the simple stuff, and systemd timers for things that requires conditions cron can't handle, such as e.g. advanced preconditions.
Then:
> Then run [...] Then try [...]
I believe that last time I tried, systemd had different semantics (no mask/unmask, etc), but memory might fail me here.
Thing is, I don't want to do any of those things. It is not looking healthy to me that a simple unix instance needs dancing around to execute /etc/rc.local. The only solution that looks sane to me is "if /etc/rc.local is present and executable - run it". Yeah, I can understand the security implications of that.
> but nothing stops you from also running a cron
For now. Apple already deprecated and banned cron from running on macOS instances in favor of their absolutely batshit insane launchd (which was the inspiration for systemd, hence the inherited brain damage).
Before long Ubuntu will ship with no cron installed and then just a bit later it won't even be available in packages. Not even as a rust rewrite.
Only if you're running a distro that lacks basic defaults from Systemd. Are you a distro like that? Have you tried just creating /etc/rc.local and seen if it gets executed?
It's not systemd's fault if your distro ships with a broken systemd install.
> "if /etc/rc.local is present and executable - run it". Yeah, I can understand the security implications of that.
That is indeed what happens if you're on a distro with a non-broken systemd install.
> Before long Ubuntu will ship with no cron installed and then just a bit later it won't even be available in packages. Not even as a rust rewrite.
Then don't use Ubuntu. Still not systemd's fault.
Technically yes. Realistically systemd fosters insane decisions like ssh socket activation or killing of cron by providing features. Useless, unnecessary, unwanted, unrequested features that solves no real problems.
Wait until they start from scratch by rewriting everything in Rust. This absolutely will happen one day.
the idea here is to make it easy for x-based wms to keep working like they always have!
It also has the benefit that if it gets enough traction, then you can displace the backend off of Wayland and go directly to hardware.
Killing Wayland would just be a bonus...
An X server selectively deprecating older functionality can be lean. There are lots of the protocol you can just ignore, and what's left is fairly easy to implement. Time consuming, but easy.
Is that actually true?
The big problem is that Xlib uses a lot of the dusty corners and the people using X11 apps expect those dusty corners to work. And Xlib is very, very, very badly architected in many places and you still have to do something sane or the apps won't work.
If you get Xcb right, you at least pick up Qt and that gets you probably 80+% of the modern apps.
It really depends on your personal needs, so your mileage may wary, but having run xtruss on most of the X11 programs I use: It's true for me at least.
The key thing is that most modern clients (and Xlib vs. Xcb doesn't matter much in this respect) use exceedingly few features of the protocol.
E.g. you'll find very few modern clients that actually uses the X protocol to draw lines with any complex options. You'll also find very few attempts at using exotic visuals, because most X servers won't support them anyway.
To take a more extreme example: You could probably get away with just ripping out most or all server-side font-rendering support. Most modern X apps use Xrender, and render the glyphs client side and creat a glyph set via Xrender or just render the whole buffer client side.
Removing those will break apps, including some that some people care about (like xterm), but API surface is similar enough that it'd be reatively easy to fix the few apps people care about, or just providing an optional shim for Xlib as a temporary measure.
For own my own use, if I were to write an X server to meet my needs, I'd just not support it - I've checked with xtruss and none of the apps I care about rely on server-side fonts at all, and none of them use any drawing primitives but the most basic line drawing and rectangles either, and all of them are capable of handling an X server that refuses anything but truecolor visuals.
I could rip out large parts of the drawing code from Xorg and all the apps I run would still work just fine...
For other people, of course, this would be a different matter, and that's fine. But I think moving a bunch of that code to an Xlib-compatible library to handle client side as a means to carry over legacy apps would be an option as a means to let people evolve leaner X servers and move shift the hassle of managing legacy code to those who care about the legacy apps.
> [wayback] is intended to eventually replace the classic X.Org server, thus reducing maintenance burden of X11 applications, but a lot of work needs to be done first.
I mean, no one puts a gun against your head to use Wayland, X will be on life support for decades and will likely work without any issue.
But with this stance, no evolution could ever happen and every change would be automatically "bad". Sure, changes have a downside of course, but that shouldn't deter us in every case.
X can evolve both by extensions and by "pseudo extensions" that effectively tell the server it's okay to make breaking changes to the protocol. There are also plenty of changes you could just make and accept the breakage because the clients it's break are limited.
I don't mind breaking changes if they actually bring me benefits I care about, but to me Wayland is all downside and no upsides that matter to me.
"People who hate person X are often a masterclass in finding fault in wonderful, intelligent, faithful, generous men."
People who talk like this are worse than systemd.
Maybe systemd (service management, logind, the DNS resolver, the loggig system, etc) will eventually be re-implemented in a way that does not have the irritating properties of the original systemd.
/* I'd say that systemd feels like typical corporate software. It has a ton of features to check all the requisite boxes, it's not very ergonomic, it does things the way authors wanted and could sell to the corporate customers (who are not the end users), not the way end users prefer. It also used to be bug-ridden for quite some time, after having been pushed upon users. It comes from Red Hat (which is now a part of IBM), so you could say: here's why! But, say, podman also comes from Red Hat, and does not feel like such corporate software; to the contrary, end users enjoy it. */
One the topic or an eventual successor, one of the aspects of systemd is how intertwined all of it is.
It’s really hard to replace just one part without either (1) replacing it with another which the same general design, or (2) remove all of systemd entirely.
Take what op is taking about for example. Instead of relying on small, reusable components for sanboxing, it just re-implement all of it into the monolithic service_manager+service_supervisor_sandboxer+…
You can’t even think of replacing the service supervisor or the logging subsystem without replacing all the other parts.
Of course, I won’t deny that this design has its upsides: this monolithic approach ships a lot faster than trying to integrate different components to work together.
And let's not forget that the author of systemd is very well known for his arrogance and explicit hatred towards unix philosophy.
Like imagine a list where we can create a form where people can give them and give reasonings or just something.
What if I can create a github repo and issues regarding this so that we can discuss about them and I can create a website later if it interests but its a really nice thought experiment.
Are we talking more about uh every software including proprietory too?
Are we talking about lets say websites too or services (what if we extend it to services like websites or even products outside of software niche into things beyond too, that's interesting too)
Another interesting point that comes to my mind might be that cryptocoins might be the lowest inverse to this software project in the sense that I believe that there was very little net positive done to all humanity in general, sure the privacy aspects are nice but still, its not worth having people invest their life savings into it thinking that its going to 100x y'know, I have created a whole article about it being frustated by this idea people think regarding crypto as an investment when it could very well be a crypto"currency" but that's a yap for another day.
I really nerded over this and I think I loved it, we need a really good discussion about it :>
Sorry if I wasn't being clear.
I am all for stable cryptocurrencies but not cryptocurrencies which prey on desperate people wanting 100x returns or promising too much that we all know is BS for hype and not delivering on it.
I myself am a teenager and uh had gotten some 100 ish dollar from winning competitions in crypto space (not that much, but I am proud of it kinda money) and uh I couldn't really get that money if it wasn't for crypto y'know.
If I hadn't made my stance clear, I am all for stable cryptocoins but that is just a very (minor?) part of the amount of things on crypto and all others usually generally harm, sure there are some outliers but here's an article that I had made when I ragebaited so hard one day about some crypto thing that I basically just wrote an article trying to explain my situation
https://justforhn.mataroa.blog/blog/most-crypto-is-doomed-to...
"This whole space is still full of scam. 99% of the times everyone has ulterior incentive (to earn money) but still I mean, it kinda exists and I mean still crypto (stablecoins?) are the only sane way to transfer money without kyc"
This is one of the comments that I had written and I hope I can make my stance clear. I just searched and found that I have written about stablecoins 10 times almost seperately in the article and uh I am all for stablecoin and I think even In my original comment, I may have said that stablecoins are the only thing that is remotely nice about cryptocoins,maybe monero if we are really streching it.
I saw a r/cryptocurrency poll of someone saying that they are 97% for the money and 3% for tech and that point this is almost like polymarket gambling except being even worse and more out of your control, almost giving someone your money if you invest in some memecoin
I kinda (like?) stablecoins (even gold like paxg is good) but not much anything else as a baseline of token for the most part, i myself have usdc in things like stellar/polygon which also have low gas fees for the most part
So what are your thoughts? I hope I can make my stance clear lol,
https://nanotimestamps.org/appseed (scroll to the end otherwise its all for appseed which was some something that I vibe coded lol)
uh, the point I am trying to make is that sure you might invest into solana for less gas fees but the point is that, you used it with usdc, I am not sure if solana requires some minimum balance of solana or smth, i think it does but just because people can pay less fees while paying usdc on solana chain doesn't make me believe that solana's native token price should increase if the gas fees are low...
Like, quite frankly, they are all driven on speculation. Sure there is some aspect of investment but the reason I believe in stock markets or even money is that they grow because people get somewhat more productive over time from all agreggates whether its through tech or just innovation or just learning from mistakes.
Crypto might grow and it might not grow, its certainly more risky and risky might make it more profitable or loss at the same time.
See the thing is, I believe that things are for the most part kinda accurate in their prices and if they aren't, then I shouldn't want to mess around trying to prove the market wrong. Because mostly its kinda an efficient market with robots that can trade in milliseconds but it was efficient even before that.
I am sure that solana's price is effectively weighed in and if its not, then well, i still wouldn't want +/- 30%
Stripe is also building its own chain (tempo) i think.. I am all for stripe having cryptocurrency too but just stablecoin.
Uh, idk man, I see these people making 30% returns and I think wow, but then I see the amount that they put and the time they invest in and they say that they are "learning" which is great but still, my questions is if this is even smth that you can "learn"
Because the thing is that if you can guarantee me 30% returns for 30 years or even guaranteed 30% or even some previous data to back that on historically, that would be great but even historical data isn't enough sometimes which is why I think of productivity as the final benchmark
My brother had like 1$ in polymarket will btc go up or down, he made 100% in an hour, he gave me all the stats of the world of these max smth curves and how they go and I appreciate my brother a lottt but I can't help but I just don't like this where he feels like he needs to earn money on top of money.
Idk maybe its me but i literally can't predict if solana can get hacked tomorrow and it can make it all 0. I saw some project on HN about how Sui, a literal billion dollar coin is suffering through some sort of not well setup nodes and a malicious actor could thereotically stop the byzantine consensus and might even bring the network down lol.
Maybe I am old schooled when I am a literal teenager but here's almost a pipeline that I feel like "investors" get into,
investing -> trading -> options trading/derivatives / forex -> cryptocurrencies -> memecoins -> polymarket trading / crypto trading
I have thought of actually writing something of my own coin with low fees with just slightly more programmability than nano to integrate it into something like cosmic while having maybe 0 gas fees but honestly, I would do it for the tech not for the money while the money is lucrative too.
If you really want, there are some better ways like how there was this 2048 thingy launched by some crypto that I kinda won for like 0 fees and got like 100 bucks and i had invested nothing in but i think that it ended.
I feel like the only times I made money was when I was lucky but if I am honest, I kinda made like infinity percent the first few times just doing smth i like / messing around but that's not really sustainable or predictable but neither is the 30% imo.
Also uh yeah some coins might require some basecoin like stellar required 0.5 stellar and 1 to get usdc but i doubt that just because of them, smth like stellars price can grow & there is this usdc project whose aim is to make gas fees also in usdc and I hope that those gas fees could be low as that could be extremely useful imo.
To each their own, I don't even trust S&P 500 at this point given how concentrated they are into AI.
Would love it if you could mail me (see my about me!) . I love talking about it!
I hope you can read the article that I shared and I would love hearing your thoughts.
The tech is cool but people have ulterior incentives. When I say not to & to invest in something as boring as world index funds, i legit don't have a ulterior motive for the most part aside from bringing what I believe a reasonable financial opinion to everybody and hearing them out.
I feel as if I currently made a gist but I might make some fediverse or reddit or bluesky posts too in the future.
[1]: https://gist.github.com/SerJaimeLannister/36b4cdc7e9bb790929...
Also side note but didn't knew that you couldn't really name gists or atleast I might have the skill issues to not find a way to change the gist name from this 36b4 thing to something better so that it might be search indexed too but idk lol
I mean, I just thought of a way of curating the software list/discussion without bloating hackernews which might make it a bit difficult to find if someone is looking for something similar imo, i can make a ask hn too but I am more than willing for some suggestions if you have any about it :p
Yes, it has a higher learning-curve than incrementally extending your constantly growing deploy.sh-script and there are many moments when it's complex and overkill. When you really need it though, no amount of in-house sysadmin-scripts will cover the same functionality with the same quality. The discussions about it online tend to have a very vocal majority of people from the first bucket, not yet realizing that they are slowly growing into the second.
All that said, it's by no means perfect and some critique is well-deserved, just that a lot of the hate comes from armchair-experts who compare it to running things locally on your laptop.
Very similar in fact to systemd, seen in isolation from an application-developer, it's one more thing to learn getting in your way. Seen from the complete system-administrators point of view, it's a consistent way to manage and secure your fleet.
It is there but most people don't care, C++ is fine imo, I mean whatever works-> works.
Its very little hate compared to systemd imo
This is incredible, really.
systemd-resolved is easily the worst piece that came out of systemd crowd. I see systemd-resolved as a transparent film that protects the shiny parts of new devices: something that is meant to be removed immediately after unboxing. This, and timesyncd.
Now, it's no wonder systemd crowd now want this thing mandatory. They truly want to be noticed and the best way to be noticed is to make someone suffer while virtue signaling righteousness.
But please do tell us why it's garbage
The "why" is important.
Today I have learned about PrivateTmp and added that thing to the evergrowing list of things to get rid of on fresh Debian instances. Yes, I did think for ten seconds and I am pretty sure I don't want that at all at any of the servers I am managing.
As for systemd-resolved, it was broken beyond repairability many years ago right at the start. At first I have tried to read manuals and play along with it. All I needed was OS using my resolvers only. Simple as "nameserver 1.1.1.1" in resolv.conf. Unfortunately, no matter how much I read the documentation, I couldn't make it do that one simple job. Oh, and manually editing resolve.conf was not an option as well because it randomly overwrote it. On a server with a static ip, mind you, no dhcp. Maybe my memory is serving me wrong but I think systemd-resolved was so bad that even +x on /etc/resolv.conf did not work, it would still overwrite it. Incredible piece of shit, just incredible. I am now removing it from all instances by default for years and I refuse to learn whether they made good or not. Clearly systemd-resolved provides negative value to the world.
I can't really avoid systemd altogether, and I like parts of it. But I also dislike that it does so much, and wish I could find a list of all its non-essential components and a guide for slimming it down and disabling any surprising or bad behavior (like its default name servers, for example).
[1] https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-ac...
The article is informative. Even a bit bland when it comes to opinion on the matter of systemd as a whole. The article is literally just saying "if you write services, complain loudly and with context about permission errors" and "if you use systemD with hardening enabled, consider it alongside discretionary access control and mandatory access control when troubleshooting permissions errors".
I've don't remember the last time I run a daemon by hand (that I wasn't developed it myself). I always just run the systemd unit via systemctl and debug that.
> A standard thing I do when troubleshooting a chain of programs executing programs executing programs is to shim in diagnostics that dump information to /tmp.
This seems like a very esoteric case in the days of structured logging and log levels.
> A mailer usually can't really tell the difference between 'no one has .forward files' and 'I'm mysteriously not able to see people's home directories to find .forward files in them'
Obviously a daemon that should access files in people's home directories shouldn't have ProtectHome=true. It's the responsibility of the daemon developer or the package maintainer to set appropriate flags based on what the daemon does. Someone had to explicitly write "ProtectHome=true". It's not the default, and it doesn't just appear in the service file.
When in doubt don't set security options at all, instead of shipping a broken daemon that you don't understand why it doesn't work.
Note: please base your daemon on D-Bus or a socket in /run and not on reading arbitrary files from my home directory.
I also don't understand the larger perspective? Should we not make our daemon run in more secure environments?
Are those days in the same room we are? Because last I have checked, logging is generally implemented as a stream of text messages with a very few bits of structured metadata. Like, almost in every piece of software in existence.
> log levels.
Those exist for decades and were invented long before anything that could be considered "structured logging".
I want systemd to get out of my lawn and stop adding features. It's not the job of /sbin/init to setup specifics of the secure environment. I want my init to launch daemons and that's it.
(there goes my karma)
systemctl edit foo.service
Then run
systemd-analyze security foo.service
This stuff is surprisingly quite powerful and it's caused me to go on a systemd binge for the last year. You can restrict access to basically anything, effectively giving you near container like restrictions. You can restrict paths (and permissions to those paths), executables, what conditions will cause the service to start/stop, how much CPU and memory usage a service can grab, and all sorts of things. That last one can be really helpful when you have a service that is being too greedy, giving you effectively an extremely lightweight container.
If you want to go further, then look into systemd-nspawn and systemd-vmspawn. This will give you a "chroot on steroids" (lightweight container, which can be used in place of things like docker) and a full fledged VM, respectively. Both of these have two complementary commands: machinectl and importctl.
`machinectl` will allow you to make these things act just like services, even allowing you to define conditions in which they can autoactivate.
`importctl` gives you the ability to download, import, and export these machines as images.
Finally, note that all these have a `--user` flag, so you don't always need to use sudo. That itself gives a big advantage because you're automatically limiting capabilities by running it as a user service. No more fucking around with groups and all that.
Honestly, the biggest downside of systemd is the lack of documentation and the lack of wider adoption and examples to pull from. But that's a solvable problem and the exact kinda thing HN can help solve (this comment is even doing some of that as well as some others in this thread). Systemd is far from perfect and there's without a doubt a good learning curve (i.e. lack of docs), but IME it's been worth the effort.
[0] https://gist.github.com/ageis/f5595e59b1cddb1513d1b425a323db...
That seems like an unnecessary foot-gun. Needing a special command to safely edit what appears to be an ordinary config file indicates some other part of the system is trying to be too clever. In this case, I think it's a combination of filling /etc with symlinks to stuff in /usr, and package managers casually overwriting stuff in /usr without the careful handling that stuff in /etc deserves.
BUT have you ever had configs overwritten by an update? Have you ever found an update to break your config? These are really quite annoying problems to deal with. Having the override file basically means you can keep the maintainer's config as your "gold standard" and then edit it without worrying of fucking things up.
This is the difference to me:
systemctl edit:
It is clearly defined what you changed. Your changes will not be overwritten by an update.
directly editing config:
File may change with update. You won't be notified of said change. If you didn't write down what you're changes were, you need to redo that work and figure it out all over again.
I run gentoo on most of my linux machines. I don't have a problem with files in /etc being silently overwritten by the package manager. Gentoo's solution isn't perfect, but at least it gets at the heart of the real problem.
I think perhaps systemd should use its clout to demand a sane solution to this problem, instead of just developing their own workaround that requires users to learn another unnecessary custom way to edit a plain text file.
There's no need to use systemctl edit to make or edit an override file, its just a convenience shortcut
Do you think anyone who's reached this point in the thread has missed out on that? Yes, obviously those steps can be done manually. But that's not what's being recommended here, and it's more steps to learn and remember, and it doesn't really matter whether accomplishing this task requires one extra command or several when it either shouldn't need to be done at all, or should work the same for any config file that needs to be protected, not just systemd service definitions.
It's almost like one of the systemd developers looked at visudo and forgot that sudo should probably never be used as an example of the right way to do things.
It absolutely is about systemd. It's a project that's famous for having almost no limits to its scope, and is highly willing to break existing conventions in order to achieve a more sensible overall system architecture. Having a specialized (and relatively undiscoverable) procedure for safely editing config files is the kind of ugly hack I'd expect systemd to be strongly against. As far as I can tell, the hack in systemd's case seems to be motivated entirely by wanting to avoid problems caused by poorly-behaved package managers. It would be entirely in-character for systemd devs to just tell people to fix their package managers, and that would be the better outcome in the long run. So I'm puzzled why any systemd proponent wouldn't regard this status quo as a wart that needs to be on their roadmap to fix properly.
An alias isn't a hack it's an... alias. You can edit the config directly as well as you can edit an override directly. Overrides aren't even a necessary thing.
But all we've done is gone in a circle. So I'm not sure if I'm talking to an llm or if you're not okay. Because you're clearly not reading responses so which is it?
> I'm not complaining at all that there's a way to make override files.
> I'm pointing out that the need for making override files
>> You can always edit the service directly. There's nothing stopping you from doing that, and most people probably do it this way.
> I think you missed the point entirely.No, please put a little more thought into this. I can't tell if you don't recognize the problem with the status quo, or just can't imagine that the problem is solvable. Your proposed choices are:
1. Learn to use `systemctl edit` to safely edit a certain class of config files
2. Manually take the extra steps of making an override file that you can safely edit with normal file-editing procedures
or
3. Directly edit the systemd config files, and have your edits silently reverted at some point in the future by software updates
The problem with this situation is that #3 is the simple and obvious way to interact with config files, but it's the wrong way. And even though systemctl will detect when you've edited one of those files and warn you to run `systemctl daemon-reload`, I can't recall ever seeing that warning mention the possibility that you may have edited your config files the unsafe way and that the edits you made are at risk of being reverted.
What I'm asking for is an option 4: directly edit the config files, and not need to worry about them being silently overwritten or reverted.
I hope you can understand that this is not one of the options you've offered, but that it would provide a superior user experience to any of the other three options. And since I've already named one Linux distro that's solved at least part of this problem, we know that it's not intractable.
Option 4 doesn't exist.
There is no system that allows you to edit system config files where you don't risk them being overwritten in an update. Perhaps you're thinking of rc files or user configs like you'd have in vim or something? But those are equivalent to overrides in this case where editing the main config would be equivalent to editing the vim default configs.
I think that's where you're confused? Thinking there's only a single config? But we're working with system level applications, not user. So the override is what I'm guessing you're thinking a config is.
Again, systemctl edit is an alias
Which is then happily ignored by systemd no matter what you do including rebooting. No thank you, it's edit the main unit and then chattr +x on it, because otherwise systemd crowd WILL break your setup on update because reasons.
What I want systemd to do is to do one thing and do it sell - launch services on boot.
What systemd crowd want is to destroy as many unix foundational tools as possible, starting from cron and logging and up to userland tools like sudo.
Pretty much no service should use that. We have ran services without that for decades until LP crowd came and said we should.
We've run services like that for decades but that doesn't mean there aren't problems with it.
https://salvatoresecurity.com/the-many-perils-of-tmp/
https://www.redhat.com/en/blog/new-red-hat-enterprise-linux-...
Also, you have like 4 replies to my comment and they're all made at roughly the same time and 2 are identical. How do you even do that? Please don't spam
True. But at the same time this also doesn't mean that multiplying tmps are going to solve those problems while not introducing quite a lot of others.
> you have like 4 replies to my comment
I did that as a courtesy to readers and you so that we can have different disagreements on different topics that you have covered in the original comment.
> tmps
If anything, I'd say that /tmp is more of a hacky solution than private tmps. Private temps is really just allowing you to force a program to perform to use mktemp rather than rely on /tmp. Much easier to provide that "patch" than to get a maintainer to modify what they're doing.
I agree that it is a more powerful tool than is needed to solve the actual problem, but a lot of times you don't actually have the power to solve the root problem (or solve it and keep it solved). So the solution works out pretty well imo.
I'm open to better solutions but you've only made complaints and offered no alternatives. Maybe you have better solutions, but who's going to know what they are and how are we going to know they're better if you make us guess? It's also why I'm not going to "fight" back because there's not a point when there's no real opposition. I wouldn't even know what I'm "fighting against" other than "the way status quo" which has obvious errors and is why this tooling was created in the first place. But please, I honestly do want to hear better solutions so I can make effective use of them.
Is it truly? I have heart some other concerns that have been raised by some developers.
> and the lack of wider adoption
How wide would you like the adoption of systemd projects to be? Like, obviously, sudo has to go because obviously sudo is not enough and LP crowd crafted their own tool; an overwhelming victory has been achieved over cron and syslogd. DNS resolution would soon be mandatory via systemd-resolved and godforbid you stay on chrony or ntpd for time sync.
What's next? What wider adoption would you like to see? Kill docker?
> solvable problem and the exact kinda thing HN can help solve
I sense there are people on HN who think differently and would definitely want to evangelize in the opposite direction.
Here's an alternative title for this post: these days, two lines in a systemd service file can easily constrain arbitrary applications to just the files and resources they need, and only those.
You can always use a simpler init system if you want
tl;dr: systemd isn't meant to be an init system, it's meant to manage services, and the alternative world where you don't have a unified system for managing services and events actually sucks.
embrace and extinguish unix philosophy.
> the alternative world where [] actually sucks.
You should bear in mind that there are plenty of people here on HN who have had the experience of running pre-systemd unixes and they know exactly how much "the alternative world" sucked. Spoiler: not much.
I don't really care to argue about it.
And worst: want to understand what the actual security properties of your policy are? The answer is buried very, very deep.
You do realize distros can also change SysV shell scripts in ways that break your use case as well, right?
Systemd didn’t “break” anything at all here. This author’s arcane debugging workflow doesn’t work for certain units who have opted into the new security mechanisms. But that is hardly systemd’s fault.
systemd introduced optional socket activation. Boom: sshd in process list is faked on Ubuntu 22+.
systemd introduced optional structured logging. Voila: every distro now ships with unremovable journald.
systemd brand introduced optional systemd-resolved, easily the worst project, and soon it's going to be mandatory for DNS resolution.
Whatever systemd makes optional, practically becomes mandatory to use in distros because systemd developers do not code optional things for them to go unnoticed.
https://cscie2x.dce.harvard.edu/hw/ch01s06.html
systemd seems to violate this at every turn, but sadly they are not the only ones to do that in Linux. I wonder at what point we should start considering Linux to be its own thing instead of "UN*X like".
you could break this out into individually named projects, but nobody complains about coreutils enabling dozens of different workflows.
systemd crown does everything for that to be technically legally right while also making sure that and modern Linux distro is unusable without a bunch of systemd-* projects.
We should be discussing management of the metastases, because systemd people still exist and they still want attention and suffering and thus they are still looking for existing AND working tools to destroy.
I don't think anyone has a problem with dissent, but that energy should be channeled into making something. Complaining about it the comments doesn't write the classic style linux distro that y'all clearly yearn for. Get coding!
cron worked just fine for decades until systemd crowd decided to "get coding".