Hacker News Reader

Comment by newsoftheday | Hacker News Reader

> fallback to running when the system comes online.

That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.

I prefer the deterministic behavior of cron, the script will run when it is specified to run, as you said earlier, as long as the system is running; and as I stated in a separate comment, it will run @reboot if I need it to run then.

> With @hourly I lose this control and multiple machines could potentially trigger backups at the same time

Then don't use @hourly, use staggered times, it's very easy.

View on HN

> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.

There are two options to fix it;

Disable persist so no catching up on missing scripts. Set OnBoot=5m so it gets ran 5 minutes after boot, so your script (say backup) is ran on boot first, then every time on schedule

Enable persist but just add sleep in ExecStartPre - very "cron" way but there is just no in-systemd option to enable "catch up" script to be delayed

Sadly no option to "run catch-up timers with delay" at least yet

> Then don't use @hourly, use staggered times, it's very easy.

Not in cron. In systemd it's just RandomizedOffsetSec=30m and it is "stable" - same host with same job will always have same delay so on multiple hosts it is spread nicely. There is also non-stable version

> That isn't something I'd want to happen, it sounds like it creates a potential queue of scripts that will flood the system on start, if it works the way you described.

This isn't what happens. If you leave it offline for days it'll only trigger the service only a single time.

I interpreted it more like "I have these 500 different cronjobs all spread out across $unit_of_time. If the system is down for longer than $unit_of_time and then comes back, does all 500 jobs start running instantly (since they missed their previous deadline)?"

Just to be clear, this isn't default systemd timer behaviour, you need to opt in by setting Persistent=true. If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool because at that scale you'd surely need better observability

You could implement this with a gitlab instance in a separate system, like two VMs in proxmox or two physical machines, and a shell executor running in them. Gitlab CI has a nice feature to limit concurrency by using resource groups. Say you have 500 jobs spread through the day and the system stays offline for a while, when it comes online it'll start processing the jobs, but will only run a single one at a time. You get visibility, logs, queue monitoring and an API to query data.

> If you have hundreds of jobs like this you need a proper queue and neither cronie nor systemd is the right tool

Eh sometimes, but you can get pretty far with one of two approaches:

1. Careful use of Requires= and Wants= to group your scripts into chains of jobs, which achieves fixed parallel (though at 100s of jobs, I hope you're generating those unit files with a tool like Puppet or https://github.com/karlicoss/dron or something and not doing this by hand).

2. Even better, just use a lockfile. `ExecStart="flock -F $TMPDIR/mylock <command>"` is pretty hard to beat. Use -F so as not to confuse KillMode and resource accounting and you're golden. Just don't use flock(1) timeouts; let systemd handle that. Heck, if you have that many cron jobs, you should be doing this even if you don't use systemd; otherwise job latency changes can cause reboot-style thundering herds out of the blue.

If you need semaphore behavior and still don't want a real job queue, waitlock (https://github.com/bigattichouse/waitlock) and many other CLIs have you covered.

1. This is spread across 500 files, maintainability goes out the window

2. If this for some reason fails, misconfiguration or unexpected shutdown, you could have a failure that's hard to track or debug

These are fine with a few services chained together, but this requires a shallow depth of dendencies. To have these theoretical hundreds of jobs chained together like this isn't practical or safe.

If you have 100 different jobs that were supposed to run over the past week, but didn't because offline, when you restart, they they all flood the system on start.

100 jobs all running at different times throughout the week is a very different load than them all falling back and running at the same time on system boot.

I don't, it's a single backup service.

Reply on news.ycombinator.com