Working with Systemd Timers
yieldcode.blog
yieldcode.blog
They are, in all ways that I care about, simply better than cron, and especially in NixOS they're really easy to set up.
Systemd makes so so so many incredibly good service administration capabilities easy & on tap. The main thing I want from systemd haters is to show how how how folks did things well before. Mostly I think this is a rebellion of the losers, people who hate having to do a good job, people who hate learning the many many many many ways we could run services better, who valorize each service figuring out their own unique special bespoke ways of running their stuff. Its fallen as fuck. We need need need the systemd socialization of running stuff well, controlling permissions & access. There's near zero precedent from the past of anyone else being responsible. That's just the situation. Systemd has drastically elevated what sysops can do, orthogonally to the service by service init launching of the past that got us no where.
https://web.archive.org/web/20250310201357/https://nosystemd...
In general it seems it's just a collection of GitHub issues. The FQDNS thing seem to be people outraged at systemd for something that the Linux kernel does not (or did not?) do well with certain hostnames -- those ending with a dot.
And there are some security issues / CVEs which is regrettable and I hate it but again, show me software that does not have them.
Seems like a very petty way to just attack something from a cursory glance. The author(s) might have as well made a ClickHouse remote query directed at systemd's GitHub issue tracker. ¯\_(ツ)_/¯
Most goals achievable in 2025 are those achievable in 1995.Its a small problem space without 30-60 years of headroom to explore and by simple logic most interesting things are going to be outside this fairly basic space.
It provided greater standardization and more power like 5 other options many of which existed prior with a simple syntax which is not without virtue.
Its just weird to refer to proponents of alternatives as "losers" who hate learning things when most proponents know systemd and other options.
Checking service status and managing services is also significantly better with systemd than any other init system I've used, and I've used a lot over 15 years of systems administration.
I still find journald a bit annoying, though, and I still am not completely sold on binary log files. I understand the benefits, but working with them is still much more opaque than text logs.
I've come to like it. At first, probably around 15 years ago, I was kind of annoyed at it and just wanted my old /var/log/syslog back. But it has some really nice features, especially for log processing and tooling, like "show me logs from the last minute" (--since) or "show me logs from where I left off" (--cursor).
As an example, this is a systemd timer I have that periodically runs rclone to sync photos to Backblaze:
systemd.services.rclone-photos-sync = {
serviceConfig.Type = "oneshot";
path = [ pkgs.rclone ];
script = ''
rclone \
--config ${config.age.secrets."rclone.conf".path} \
--bwlimit 20M --transfers 16 \
sync /mnt/photos/originals/ photos:
'';
unitConfig = {
RequiresMountsFor = "/mnt/photos";
};
};
systemd.timers.rclone-photos-sync = {
timerConfig = {
# Every 2 hours.
OnCalendar = "00/2:00:00";
RandomizedDelaySec = "5m";
Persistent = true;
Unit = "rclone-photos-sync.service";
};
partOf = [ "rclone-photos-sync.service" ];
wantedBy = [ "timers.target" ];
};But with NixOS and this example specifically, you also get (at a minimum):
a) A declarative config without requiring an external tool like Ansible.
b) Any dependencies (rclone in this case) will be implicitly installed if not already present on the system.
c) If configuring a remote machine, this will copy over the encrypted rclone.conf file and decrypt it on the target.
d) And of course, it's trivial to version control and track changes to the config over time.
In the meantime I'm just going to keep using void because I know how to fix it when it breaks. Systemd i have to google how to even find the service logs.... they certainly aren't easy to find via the filesystem any more.
Finding logs for a specific service is a one-liner:
journalctl -u rclone-photos-sync.service
Here is a nice overview: https://www.digitalocean.com/community/tutorials/how-to-use-...
Do we expect an average dude to just pop into a cockpit and land a plane with no previous training?
And I agree that NixOS without Flakes is a worse experience overall.
The really nice thing about systemd timers is that as units you can get status of them and check the journal output the same as any other unit.
the time when the service unit was last triggered is stored on disk. When the timer is activated, the service unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive. Such triggering is nonetheless subject to the delay imposed by RandomizedDelaySec=. This is useful to catch up on missed runs of the service when the system was powered down.Does anybody else have this problem? How do you get over it? (it seems like such a ridiculous reason to avoid picking up an otherwise good tech)
I started making my own packages for Nixpkgs, and I have made dozens (maybe hundreds) of flakes, so I'm just kind of used to the language now. It's an ugly language, but once you write a lot of it, you get used to its quirks.
And there are some niceties, like being able to define a variable and use that variable to access and define fields, kind of like JavaScript:
{
blah = {
${myVar} = "something";
};
}
Otherwise it's just kind of an awkward functional language. You get used to it.
But I don't want to have to run it or even think about it. Lennart has other plans, hence his campaign to get distros to require it and software up the stack to hard-depend on it. I don't like its design, I want to opt out of it in favor of something better, but one of the development effort's goals is making that extremely difficult for modern Linux systems. That's my issue with it.
But really it macht nichs for me right now because I run Void, btw.
Others design were already tested extensively, and the one that stuck was systemd, don't you already stopped to think that the "better design" is the one that systemd are currently using?
My criteria for a good init system design do not align with Lennart's, or with Red Hat's. It's not winner-take-all in principle, even if it approaches being so in practice.
Distros requires a lot of stuff, e.g. libc. This does not seem inappropriate given what a distro is.
> software up the stack to hard-depend on it
What does this refer to?
I can see this for session/user/service management programs. But not normal single-user programs.
Having used it for several major releases of Debian technically I'm happy to have it - it's an improvement over the sysv init & crontabs. I'm not completely in love with journal compared to regular logs but I can see some advantages.
The only time I don't is when it just cannot do something, and then you need to layer the old Unix way on top of it using the wrong semantics, i.e., services that are actually complicated bash scripts which are used to do something for which systemd has specialised units, like an (auto)mount. Then it's all the pain of brittle Unix scripts, with misleading semantics, and an extra layer of abstraction and set of limitations to keep in your head.
But no, I genuinely think that systemd is great these days.
I do understand why people become irked by it becoming a dependency of programs which really should be able to function without it though.
One thing that makes it difficult is how scattered between all the different manpages the documentation is. It took me quite a lot of time and frustration to build up a decent mental model of what everything is for, when it needn't have if (ironically) the docs were just written in a less Unixy way.
But making my system feel more event-driven and less imperative has been a pleasure.
For example, show the next five trigger times for the end of the last day when the month has 31 days :
systemd-analyze calendar --iterations=5 '*-*-31 23:59:59'
We have a free tool called CronitorCLI that includes a cron-like shell. You can run and test your scripts in an environment that matches how they will be run by cron itself.
Thank you for many years of reassuring me when I doubted my cron-expressions.
And quite frankly, your experience is likely due to your personal knowledge bias. Systemd timers are quite easy to debug, in fact easier than cron in my opinion. systemctl list-timers lists all the timers, when they last ran, when they will run next, and you can use other commands to inspect each timer in detail and all associated logs.
In contrast, I find cron much harder to debug. Starting with the first problem, you must first figure out what cron is running, as different implementations have different behavior!
Although speaking of which, cron dropping privileges is not as secure as systemd running as the user before parsing the user's timers, from a defense-in-depth perspective.
Cron has no diagnostics whatsoever. Most asked question "How do I force run cron job for debugging" and answer "You don't"
You just add script and pray it works. If it doesn't work there is nothing to work with. No logs, no checks. Nothing.
It not 1-to-1 replacement. It is superior replacement. Cron is just unusable.
All the cron issues I've had boiled down to a difference in env vars. So testing the cronjob means reproducing the environment like this SO answer explains.
https://stackoverflow.com/questions/2135478/how-to-simulate-...
EDIT: yea, the email reporting is certainly missing, but it was hard to control it since whole STDOUT was shipped, which is not what I wanted most of the time anyways. It would be good to come up with some way to still have small overview emails sent about important jobs done, maybe a dependency service which starts when important job finished and just sends an email about that
Correct. If there's no `Unit=` specified in the timer unit, it defaults to the service unit of the same name. See https://www.freedesktop.org/software/systemd/man/latest/syst...
This is also a common pattern for the socket units of socket activated services
> If you want to execute pre/post commands you have to do it inside the script itself
So?
> There are no built-in logs
Every cron implementation I can remember using logs each run to syslog and emails me the output of the run by default
> There is no built-in status monitoring
I can't think of any built-in status monitoring that systemd has for timers that's materially different from cron's logging/emailing
> If the system is down when the cron needs to run, the cron will be missed
Some cron implementations support this and some don't. Most modern ones that I'm aware of do.
Much more significantly, the amount of setup involved in a systemd timer is way higher than putting a line in a crontab, especially for the author's case of just running a backup script.
Cron only involves running `crontab -e` and adding the line "@daily /path/to/script.sh" (which also handles the author's issue of cron "skipping" runs if the system was powered off, assuming the cron implementation uses something modern like anacron)
Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.
Need to distribute lots of cron jobs evenly to avoid overloading the system? Use RandomizedDelaySec.
Some cron jobs are flaky and you want to re-run if it fails? Add Restart=on-failure to corresponding service.
Some cron jobs conflict with each other? Set Conflicts=foo.service or maybe Before & After.
Sure, all above are possible with shell scripts. But systemd provides a standard, reliable way to define these properties.
Yes, they do. And as the very first sentence in my comment says, I like systemd timers when they're appropriate. None of the features you've mentioned were used in the author's solution, which is the sole thing I'm arguing against.
Counterpoint: this requires root, while you can edit and run systemd timers without root. Thus it's also generally more secure, both when creating the job and every time it runs.
> Systemd involves writing a 7 line timer unit file, an additional 5 line service file, running a daemon-reload, then enabling the timer. It turns what's usually a 10 second mindless task into a much more involved procedure. That can be worth it if there are material benefits from it, but I'm not really seeing them here.
You could write a script to automate it, if it's such a big deal. Creating timers/cronjobs isn't something that needs to be done often enough for this to matter.
If it is, another counterpoint: systemd supports creating transient timers, which you could do programmatically.
And final counterpoint: you can make systemd understand crontab and convert it into timers (systemd-crontab-generator)
Counter-counterpoint: No it doesn't? Running `crontab -e` as a non-root user will edit that user's crontab, and running it as root will edit the system crontab. Cron can be configured to deny users their own crontabs, but every common distro I'm aware of defaults to allowing user crontabs.
> You could write a script to automate it, if it's such a big deal
Or I could not bother with that and just use cron?
> another counterpoint: systemd supports creating transient timers
What is this a counterpoint to? That the author's particular use case of running a backup script once a day is a task better suited to cron than systemd timers? I'm not sure how transient timers are even relevant here, much less a counterpoint.
Running it as root will edit the root user's crontab (in /var/spool/cron), which is separate from the system crontab (/etc/crontab), which has a slightly different format.
Per Gentoo's wiki, both fcron and cronie have their own (different!) ways of whitelisting non-root users to run cronjobs.
> Per Gentoo's wiki, both fcron and cronie have their own (different!) ways of whitelisting non-root users to run cronjobs.
Both allow any user in the cron group to have their own crontabs by default, and both support optional .allow/.deny files
It's part of the Single Unix Specification.
https://pubs.opengroup.org/onlinepubs/9799919799/utilities/c...
Thanks for sharing it though. I wasn't aware it was in the posix spec and it explains why pretty much every implementation supports .allow/.deny files even when most already implement better access control mechanisms.