Show HN: Dcron – Distributed, fault tolerant job scheduling system | Hacker News Reader

Show HN: Dcron – Distributed, fault tolerant job scheduling system

dcron.io

42 pointsvictorcoder24 comments

Open articleView on HN

You may want to rename your project. dcron is Dillon's cron for quite a lot time already.

Yeah upon reading the title I thought Dillon had done some improvements on dcron.

Yup, thought the same thing.

Right I know, but if he don't mind, both projects are enough different to not create confusion.

Need more time to see if this will be necessary.

No, they are not different enough. Both are cron daemons. Both are in the end supposed to be packaged. It's really enough to make the newer one (newer by decades!) to change its name.

Honestly, it's just the classy thing to do - the impact of changing your name is nominal, the potential confusion with the other Dcron is significant.

The project has been renamed to Dkron due to respect to Dillon's cron and to avoid confusion.

Man, how I wish Kala and Dcron would just work together.

https://github.com/ajvb/kala

https://github.com/victorcoder/dcron

Fusing both would be AWESOME (I'd even like to help).

What Dcron gets right:

- Distributed by design

- Raft for the reliable parts, gossip for lightweight sync, a match made in heaven

- Linux mentality

- Dashboard included

- Good docs, nice website, looks serious

What Kala gets right:

- Built much more directly after Chronos, which has an amazing and battle tested API

- Using ISO durations instead of NIHing their own format

- Dependent jobs

While Metafora doesn't support cron-like syntax for job scheduling or tag-based node targeting (yet!), it does support distributed fault tolerant task running. We hope to build out a webui soon as well.

However since Metafora is written as a Go library and provides a statemachine implementation for task handlers to use, it requires writing your distributed tasks in Go.

https://github.com/lytics/metafora

I have a bunch of Python scripts that run a various intervals (hourly, daily, monthly, etc). I have trouble keeping track of which jobs fail -- unless I notice an email doesn't show up, or a metric is missing. Is Dcron something I would use to keep track of my cron jobs and whether or not they ran successfully?

At this point I don't need my jobs to be distributed, I'm just looking for a way to keep track of them all and visualize characteristics of the job (start time, length, successful, etc). Ideally, I'd have a way to re-run the job within the UI.

Check out Airflow (open source project from airbnb that does all of that very well).

Sure, it can do this well. It can run in a single dcron/etcd node if you don't need it to be fault tolerant. At this point you should be checking the job status in the ui as it doesn't have notifications yet.

You can also try cronitor and others like it. You add a CURL command at the end of your script that 'touches' a HTTPS endpoint. If your script doesn't check-in at pre-defined intervals, you get alerted. They are perfect for situations where you don't actually want to set up any infrastructure.

Disclosure: I work for Iron.io

If you're open to a hosted solution, check out IronWorker: http://www.iron.io/worker

It's an async task processing service with a built-in job scheduler. You can upload your python scripts to Iron.io, then set schedules and other triggers to execute on-demand. We have a dashboard to manage tasks and schedules, see what ran and what failed, and you can visualize the characteristics you're looking for. We do distribute the workloads for you, but sounds like it could be a good fit.

Hey,

I've been tinkering with this concept recently, it's not quite ready for beta testing, but I'd be happy to take feedback on what I've got so far: http://croncloud.io/

I haven't done any digging, but I'm assuming this uses etcd for membership? How does that work - how does a server know whether its a leader, and how does failover work?

Nope, it uses serf for membership.

How do I define jobs? Can they be written in any language? I didn't see anything in the documentation.

Looks like it's defined using the format you see in this json file: https://github.com/victorcoder/dcron/blob/master/job.json

The "command" field is the most important part. What you put there gets executed like this by their agent (on Linux): /bin/sh -c <your_scheduled_command_name>

I was looking into that too. Based on this job.json[0] definition from the git repo, it looks like it just executes the command. So I believe you could have the job written in any language.

But I think this also means you have to copy every job executable to every server that might have a chance of running the job.

[0] https://github.com/victorcoder/dcron/blob/master/job.json

Added to the docs, big omission, sorry! http://dcron.io/docs/api/#create-or-update-a-job

Any change that dcron could manage the job scripts itself? I.E. you store the raw command scripts in dcon, then reference them in the create a job endpoint?

For example:

    POST /scripts --data 'node /home/node/myJob.js'

That returns a script id { "script_id": "507f1f77bcf86cd799439011" }

Then:

    POST /jobs --data '{
       "name":"cron_job",
       "schedule":"@every 2s",
       "script": "507f1f77bcf86cd799439011",
       "owner":"foo",
       "owner_email":"foo@bar.com"
    }'

This way you don't have to manage/create the scripts on each node manually.

I think you missed the point, jobs are stored in etcd so they're distributed and all dcron (server) nodes will see it. You can even create jobs from any node. Note that, non server nodes doesn't need access to etcd.

Cool! Might use this when I renovate http.sh

Reply on news.ycombinator.com