Hacker News Reader

At LinkedIn we heavily use Azkaban for this. (Open source: https://azkaban.github.io/) Azkaban API can be used to launch offline computation jobs as necessary - Azkaban ensures monitoring, SLA alerting, failed restarts and other dependency management etc.

View on HN

Azkaban really seems to strike the right balance between simplicity and featurefulness, I'll definitely give it a try! Plus it seems relatively simple to deploy & maintain.

The documentation often mention Hadoop and data jobs, have you also used it for non-data things? Would you by chance have some workflows examples?

You can use this for any execution. eg. here is a job type to trigger shell command such as ' echo "hello" ' http://azkaban.github.io/azkaban/docs/latest/#command-type

Note execution environment for such jobs is Azkaban executor server itself, so you have to take care of resource management (eg. one job taking all RAM on the machine will affect other jobs running on the same machine)

Reply on news.ycombinator.com