Hacker News Reader

Comment by rb808 | Hacker News Reader

This looks great, I've always wanted something like this. I've always had autosys or controlm at work and they both suck.

I'd just prefer if it had been around longer. Any other open source alternatives out there? I only know of Airflow, k8s Cronjobs.

View on HN

There you go https://github.com/meirwah/awesome-workflow-engines

> Any other open source alternatives out there?

Here's a big handful of them: https://github.com/pditommaso/awesome-pipeline

There's also Luigi. https://github.com/spotify/luigi

I was just reading up on Broadway, written in Elixir, (https://hexdocs.pm/broadway/Broadway.html) that provides the fundamentals of batching/job control. It’s by the creator of Elixir and is based on 7 years of libraries in the area so the fundamentals are pretty well honed.

I've had alot of success using Apache NiFi as a distributed scheduler / general purpose workflow tool.

I'd love to see what kind of complexity you are managing there, and how.

The only other one I can think of off the top of my head is dagster: https://github.com/dagster-io/dagster

It’s made by Nick Schrock of graphql fame, among others. I’m sure there are 100s of these projects though.

From the beginning of Dagster's readme:

> Dagster is a system for building modern data applications.

> Combining an elegant programming model and beautiful tools, Dagster allows infrastructure engineers, data engineers, and data scientists to seamlessly collaborate to process and produce the trusted, reliable data needed in today's world.

Two paragraphs, communicating zero bits of information. I wish Github repositories, of all places, didn't contain such noninformative copy.

Reply on news.ycombinator.com