Plan – Cron jobs in Python | Hacker News Reader

Plan – Cron jobs in Python

github.com

57 pointsfsp18 comments

Here is some brainstorming on use cases.

1. You can validate cron jobs and programmatically change or reject some of them. Or maybe just offer a small set of services as a cron job.

2. Once you have a security model you can expose this as a web service and offer cron jobs over the web.

3. You could extend this API to use different implementations of cron. Instead of calling the actual cron you could implement cron in Python or maybe use a cron web service.

This would be useful incase, if you want to set cron dynamically.Based on the executed result, next step can be manipulated via this.

1. crontab -e command automatically recognize if you have a syntax problem in the newly edited file:

    $ crontab -e
    crontab: installing new crontab
    "/tmp/crontab.QhkOLb/crontab":25: bad minute
    errors in crontab file, can't install.
    Do you want to retry the same edit? (y/n)

2. See better solutions (like Celery) below.

I don't get it. What's so hard about cron jobs that you have to write a wrapper for it? It is easy to understand if you just read the crontab file comment at the top. There is just two commands:

    crontab -l

for listing

    crontab -e

for editing. What else do you need?

With this, instead of learning crontab simple syntax, you need to:

- install this package

- learn its API

- run it and still edit the crontab file :D

seems like not worth it.

Whenever gem does the same thing. So you can keep it under version control with your project.

It's just a text file, why not just add it to the version control of your project?

If it's because users can only have a single crontab, you can always use system crontabs instead or "cat" them together.

You can install a new crontab file with

    $ crontab [ -u user ] file

How about dynamic periodic tasks?

For instance, server side exponential backoff.

This approach has a human in the loop. There's plenty of use cases where you want a program to edit cron jobs.

I see the value in this: it provides a nice programmable interface to cron. Unlike what you might think at first, cron has some really interesting idiosyncrasies that you might not appreciate at first. For example, if your filename in /etc/cron.d/ has a "." character in it, it will be silently skipped [1]. Also, files in /etc/cron.d/ will not inherit environmental variables from /etc/crontab. Consider how many hours have been wasted "debugging" these issues.

That said, cron is easy to abuse as well. The example of providing exponential backoff is kind of silly: there are better tools for the job, including Celery. Cron also does not have seconds precision which is a bummer for lots of short-lived jobs. Lastly, cron only runs jobs at the specified time. If you adjust the system clock, or shut the system down, your tasks might be skipped. anacron [2] might be a better solution in some ways. Lastly, all implementations of cron that I am aware of don't have a great ability to make sure tasks actually run and finish, don't have retries built-in, and have a horrible interface for reporting errors (any output of the script is emailed to root@localhost from root@localhost).

Based on the above, you might think I am saying to never use cron. That's not at all true. Cron is extremely useful as a systems tool. If you package your software in the distro's package format (you should at least give this a try to see how the system is put together), cron is pretty much your only safe choice. Also, as long as you follow the rules cron sets up, and know of its potential issues, cron will be the workhorse that gets things done. Tasks like log rotation, periodic sanity checks, etc. are great for this. If you are writing a web application and are already using Python though, look at more sophisticated options such as Celery before using cron directly.

[1] http://manpages.ubuntu.com/manpages/maverick/en/man8/cron.8.... and https://bugs.launchpad.net/ubuntu/+source/cron/+bug/706565

[2] http://en.wikipedia.org/wiki/Anacron

And if you are already using Celery, it has crontab feature http://celery.readthedocs.org/en/latest/userguide/periodic-t...

Yeah, we are using that as well. It's nice because it hooks up to your Django admin and people with the right permissions can reschedule stuff.

I don't see any use case for Plan, it's just complicating an easy task.

I can see one use case for this family of tools: if the same script could generate either crontab files or launchd agents, so it could generate scheduling files for MacOSX, where crontab is discouraged.

MacOSX uses launchd as its crontab, but it retains the crontab interface.

Celery beat is pretty much exactly the same thing as cron except it's another service to configure and worry about keeping up and running. And it uses 35 MB of RAM instead of...well, I guess cron uses a few kilobytes.

I can't see the point of it at all.

When I want to kick off celery tasks I just have a cron job execute a django shell command.

I guess cron uses a few kilobytes.

A full megabyte on my machine, actually! Still a long way from 35MBs, though.

I'd typically deal with this by exporting the functions to be ran as entry points in the the setup.py file, and then use whatever configuration management tool I'm using (Ansible, Puppet, whatever) to manage the crontab file in /etc/cron.d. Ansible's cron module makes this especially easy.

I wrote something similar a while back, but it only requires an hourly cronjob to run a bunch of Python scripts in an arbitrary file structure: https://github.com/nlindblad/ocarina

I use it extensively myself for scraping various sites/APIs.

Reply on news.ycombinator.com