Ask HN: What job scheduler do you use? | Hacker News Reader

Ask HN: What job scheduler do you use?

26 pointspmchorus13 comments

In a previous company, I used the in-house job scheduler. Despite some bugs and lack of documentation it did what we asked for: schedule jobs according schedules and timezones, and handle dependencies.

Now I'm working for another company, we have the same need:

- Orchestrate hundreds of jobs

- Some need to start as soon as their dependencies are ready

- Others need to start according to timezone-based (and possibly dependencies)

- Notification on failures

- Make jobs fail if their dependencies are not ok after a given time

- Force a job status (i.e. set it "ok" even if it failed)

- Reasonable access control

- And if possible a nice dashboard!

While these requirements seem standard to me, I cannot find any acceptable solution:

- Cron is cool for the simplest things, nothing more

- Rundeck is nice for simple workflows, but not for complex dependencies

- I'm tinkering with SOS-Berlin Job Scheduler for days now, and I cannot find proper ways to orchestrate my dependencies

- Control-M looks over priced and certainly not deployable without months of project planing and contract negotiation

- Luidgi and Airflow are too much focused on data workflows, not on generic job scheduling

What are you using? :)

For php laravel projects are using this service:

- https://github.com/cronboy/cronboy

- https://cronboy.com/

It doesn't fit needs of @pmchorus company, but is useful for simpler cases when you need something more efficient than cron and more accurate than queue delayed jobs. (Example: we use it for sending triggered email deliveries, where batch processing of letters using cron is not possible, because of amount of letters and cron schedule limit)

The idea is that you schedule job on specified time and cronboy invokes it via http and execute it on your server.

Features:

- Seconds accuracy of execution

- Statistics on invoked jobs

- Secure execution of your code

- Retrying if job failed

- Laravel friendly facade api

Airflow might be primarily intended for workflows, but scheduling is fairly robust. But you also say that you want more than just simple Cron so I suggest taking another look.

What I really appreciate about Airflow is that it is self documenting. Anyone can logon and see the exact workflow, schedule of that workflow, the status of last few job runs and the python code that ran if they desire. Airflow documentation itself is good and getting better.

It can run a simple cron job but it also gives you room to build it out into much more. We use our Airflow scheduler to call simple API endpoints and then that same server does daily loads into our data warehouse.

Also 1.8 just dropped to stable and it is a huge improvement upon earlier versions.

Sidekiq. It's a wonderful project and has formed the basis of so many projects I have worked with over the years ever since it's inception. It's also a great example of an OSS project with a healthy commercial offering.

Given that you mention Luigi, have a look at Styx. Currently in use at Spotify as a complement to the underlying job processing frameworks.

https://github.com/spotify/styx

Looks like it only support cron expressions and no timezones. Moreoever, I don't plan to deploy a Kubernetes cluster in the short term (looks like to be a huge work to have something ready for production). I'm primarily looking for something I can install on the existing systems.

https://www.nomadproject.io is awesome for scheduling both long running processes and time triggered jobs

It does not seem to handle dependencies.

Any experience with http://www.schedulix.org/en ?

https://mesos.github.io/chronos/

I have no Mesos cluster running for now. I'm looking for something I can use on the existing infrastructure. From what I read, it does not support timezones.

Celery might be what you looking for.

look at slurm and other things based on it. Slurm plus mesos + airflow will meet all your needs.

Consider AWS lambda. It's not a monolithic system, but may work depending on your use-case.

Reply on news.ycombinator.com