Hacker News Reader

Comment by foobarian | Hacker News Reader

> Here is an example of a typical oncall experience: 1) get paged with the command line of the failed job; 2) ssh into the scheduler machine; 3) copy & paste the command line to rerun the failed job

No doubt this works if it's an established procedure, but if I was approaching a system I wasn't familiar with I would never do (3) because the environment can differ wildly between crond and a login shell. It is safer to edit the cron schedule and duplicate the entry with a time set for few minutes in the future. (And clean that up afterwards).

View on HN

Not only that, but just ssh'ing in and blindly rerunning the failed job isn't the answer anyway. Research why it failed. If the job has to write a file on a full disk, you can rerun that thing a hundred times and it will never work. I'm sure they must have missed something from their write up as I can't imagine that's their on call playbook.

You could even write a cron job to clean up the crontab

Or you could use at(1) or batch(1) to invoke the job once. That seems significantly easier and less error prone than trying to clean up crontab.

at(1) is great, but you'll still have the problem of a different environment to cron (it copies most of your environment variables and working directory).

Or use Puppet/Salt/Chef/ansible to manage the file automatically

I've seen this done previously. It gets arduous to maintain a large list of cron jobs, teach everyone how to add new ones, rebuild the cron box and redeploy on updates to config management.

Just use Jenkins, until you outgrow Jenkins. Then use a distributed scheduler.

Can you recommend one?

We recently had to implement a prototype in house, not that easy as I thought it would be, but we could not find a good solution, e.g. handle 100k jobs, long term (never expiring), short term (depends on an entity's live cycle), high available.

Thanks.

Kubernetes does Cron jobs.

Kubernetes if you're a containers first shop, Hashicorp's Nomad if you're not just containers (can run binaries, etc).

Reply on news.ycombinator.com