Cron best practices
blog.sanctum.geek.nz
blog.sanctum.geek.nz
- Where to put the logs, they just go into the system journal. Easy to check, and if you already have alerts/monitors/whatever for the journal any new job integrates trivially.
- Cron shell doing weird things.
- Easy switching on/off of each timer.
- Easy monitoring of when tasks are executed and the results.
- Jobs don't get started if there's one already running (I think)
- Better error messages when you mess the syntax.
- More possibilities for scheduling.
- Launching the job now if you need it to.
"Run this command the day before the last day of the month" - process some logs
"Run this 15mins after the system boots" - connect to things once ready etc
It is pretty cool
i.e. runs every two days at 11pm. or am i mistaken?
I'd consider offloading the work to the shell if you want to run things every other day, e.g.:
* * * * * [ $(($(date +\%w) \% 2)) -eq 0 ] || echo skipped-a-day
Top of my script was a simple if-clause, when alternate-day.txt is found, it will be removed and the script runs. If no .txt is found it will be created and the script exits. A bit lame, but it worked and was quick :)
As mentioned, the next problem appears when the job takes longer than two days, which requires using flock. I often prefer to use mkdir instead of flock, provided your mkdir is atomic and POSIX compliant... which is usually the case AFAIK. E.G. mkdir a-lock-dir || exit;... rmdir a-lock-dir
Ultimately I don't have too many complaints about cron. But, like anything, it has it's own quirks. Indeed a handy article.
There's many such examples - every 1, 2, 3, 4, 6 hours works - but every 5 hours doesn't. But every 2 days and every 14 days are requirements I've met IRL.
Which is exactly what I want! Error? I must know. Output? If I left any, then it's a notification, and I must know. No output? A maintenance task succeeded, nothing to report.
I can't remember the exact unit file syntax but you can define Before/After hooks, so you can point that to a script that emails people (or in my case pushes to a notification queue).
Absolutely.
I only randomly learnt about systemd timers thanks to a comment I read here on HN once. Since that point I've never looked back, its simply awesome.
The main reason is its enabled me to remove a whole load of boilerplate from my scripts because systemd takes care of it for me.
So my scripts are now shorter, more readable, and less prone to bugs.
> Jobs don't get started if there's one already running
Yes. This was the biggest selling point for me.
On cron, I had a ton of boilerplate using all sorts of tricks to avoid the race condition of the script being run again before it had finished its prior execution.
With systemd ? Remove the boilerplate, and, erm that's it. IIRC I don't think it even needs any special lines in your systemd unit file, it Just Works (tm).
Let's not go there shall we.
Yes I know about flock, that's what I used.
But if you are writing code correctly, you still end up with boilerplate. For a start, you need to write extra code to check the error status returned from flock.
And then there's the overall point I was making. My "ton of boilerplate" referred to the overall boilerplate, i.e. flock was just one element, so then consider all the other boilerplate that was required to do all the other stuff that systemd does out of the box.
Frankly, its a no-brainer. Systemd wins every time.
Lots of small self hosted things get written as if they were meant to be set up by IT professionals, and even then some of it is an unnecessary hassle even if it is meant for a commercial setting.
What's even worse is pre systemd stuff that would put multiple unrelated services in one file. Systemd does it right because all your services are separate files with very little global anything that isn't inherently global.
It's just like the LAMP stack. There are almost no PHP apps, just PHP code you can assemble into an app yourself with other daemons, usually with some subtle ways to get it wrong.
Wheras more recent stacks are often unzip and run or apt-get and run with only a small amount of local configuration.
Separating the schedule from the process might be an annoyance to newcomers... But I promise it's worth it
It's two files to template instead of one line, but it fixes essentially every lesson we all learned with cron. $PATH being incomplete, locking, etc
Another nice benefit is dependency handling.
If your 'cron job' requires networking you have to write boilerplate code or let it fire/fail endlessly. Pray the processes don't linger/stack or otherwise make a mess.
If a timer needs networking, set After= and Requires= to networking.target or whatever is appropriate for your distribution - and you're golden.
Even better if the job acts against another actual systemd service.
Using systemd-networkd you can tie the timers to specific interfaces. [but this handles NAT weirder than you're probably used to, if a host acts as a router prepare to adjust]
I should write a little bash script that takes a name, a command, and a schedule and just sets it all up. Has anyone already done something like that?
Of course, there is always crontab guru (E.G: https://crontab.guru/#*/12_*_*_\*), but still, it's easier to find out with a comment next to it. Besides, it documents the intent, which may shows that there is a mismatch with the reality.
(Unless cron has gotten smarter since last I used it, which was admittedly at least a decade ago.)
(whether or not you trust that is another thing...)
Local time changes of less than three hours, such as those caused by
the Daylight Saving Time changes, are handled in a special way. This
only applies to jobs that run at a specific time and jobs that run with
a granularity greater than one hour. Jobs that run more frequently are
scheduled normally.
If time was adjusted one hour forward, those jobs that would have run
in the interval that has been skipped will be run immediately. Con‐
versely, if time was adjusted backward, running the same job twice is
avoided.
Time changes of more than 3 hours are considered to be corrections to
the clock or the timezone, and the new time is used immediately.
It is possible to use different time zones for crontables. See
crontab(5) for more information.
If cron runs on more than one host, it can be useful to add some randomness to the time to avoid the same job being run on exactly the same time everywhere. Red Hat-family distributions have RANDOM_DELAY variable for this. Or just add the delay when Ansible or a similar tool installs the job from version control. (Which you use, right?)
https://wiki.archlinux.org/title/Systemd/Timers#Transient_ti...
In addition, I also want to see the status of the transient cron job and see how much time is left before it executes, like `systemctl list-timers temporary.timer` would show.
Also, you want to make sure your job has been running. I've used a selfhosted instance of healthchecks.io in order to warn me if something has not run.
At one former job where I had a lot of cron jobs, I even wrote a re-usable wrapper utility for it.
Many unix command line executables are pretty bad at it (especially if you deal with anything proprietary), so I get why people resort to just throwing away output from cronjobs, although it is a problematic lack of observability, true.
You have to use systemd timers for that... It's completely overkill (2 files each with a dozen lines) but it works much better in practice.
Using Ohmycron is better than nothing, but it doesn't handle timeouts and stale locks and some other useful things. Seems like we still need a better universal wrapper for cron jobs.
Guess I'm one of the lucky 10k today - but a search for information isn't turning up much. Could you expand on the maildir method a bit more?
Fun fact, Kubernetes cronjobs are highly-available and fault-tolerant[1]
[0] https://kubernetes.io/docs/concepts/workloads/controllers/cr...