Dalga: Job scheduler written in Go | Hacker News Reader

Dalga: Job scheduler written in Go

github.com

56 pointscenkalti14 comments

Have you looked at Rundeck? http://rundeck.org/

It has rest based interface to do everything + UI to make it easier when you need to do one off things. It allows you to control execution across multiple nodes. Good thing is you can setup your processes by tag of a node in Dev and export the project as is on Production and it will still work.

It's a runbook automation tool but can be used like in place of chronos, oozie, spark job server etc.

Why Dalga instead of plain cron?

* Dalga stores jobs in a MySQL table which is handy if you are already using MySQL (backups, etc.)

* Dalga can schedule one-off jobs.

* Dalga has built-in retry mechanism.

A tight dependency on MySQL is somewhat limiting. Have you considered making that an interface with a default implementation that uses MySQL? It pretty much ought to be a mechanical transform where you simply extract all the places you touch MySQL into a method, then write an interface around that method.

In large setups, it is usually clever to isolate job execution policy into a separate system. This simplifies a lot of contextual configuration of the system at large.

You can have cron emit the cascading event if you want but you often need a system which can "prod" other systems and ask them to do things because of a timing event. In such systems, cron often falls short as the system around it grows.

For local timing of events, cron is a really good tool however, and I wouldn't hesitate about using it there. It is for system-wide events these general job schedulers tend to help.

Why Dalga instead of of some other job scheduler? Because all service software will eventually be rewritten in golang, apparently ;). Don't hate the player.

To be honest, I like the fact that a lot of the 'old` stuff is being rewritten in Go because I find it a much easier language to grasp than C, C++ or Java. So I say rewrite them all! (just my opinion of course)

Well that's a driver for golang for sure. Additionally, while we're complaining about golang not measuring up to our programming sensibilities, the people who write us checks want 1) correct, 2) on time, and 3)in budget. If golang helps us do that better than other languages in spite of the fact that it doesn't have generics or whatever (anecdotally seems to be the case), no need to apologize for it.

Why Dalga instead of Kala? https://news.ycombinator.com/item?id=10046164 Aren't they very similar? Isn't it better to join the efforts?

Kala is in Alpha stage. Dalga is stable and used in production ~2 years at https://put.io.

So I should probably ask the same question in the other thread :)

On a completely different note, kudos for the idea of charging a small amount that gets contributed to charity for trial accounts (on https://put.io/).

Not sure if you guys came up with that, but it's the first instance I've come across. Great idea -- filters out freeloaders from potential true customers.

Does Dalga support job dependencies? I.e. job A can only run after e.g. job B and C have finished first.

No.

Reply on news.ycombinator.com