Aside from this platform, which I've never heard about until now, I'm wondering what others are using in the workflow orchestration space?
I'd assume Airflow is the most prevalent, but there's also Argo getting quite a bit of momentum lately.
I'd assume Airflow is the most prevalent, but there's also Argo getting quite a bit of momentum lately.
"My team already knows Airflow and/or I want to pay Astronomer a lot of money" -> Airflow
"I love YAML and everything is on k8s anyway" -> Argo
"I just want something that works out of the box and don't want to host my own compute" -> Shipyard, maybe Orchest
"I want a more flexible, generic workflow engine and don't care about writing orchestration in Python" -> Temporal/Cadence
"I am very nostalgic" -> Azkaban, Oozie, Luigi
"I love clunky Java solutions to data problems" -> Nifi et al
"I like to pay for half-managed solutions and late upgrades to a first-generation technology" -> AWS/GCP hosted Airflow options
"I am on AWS and it doesn't need to be complicated" -> AWS Step Functions
I know there's a degree of oversimplification going on here, but there's something to be said for having a simple bullet-list breakdown of all the use-cases - alongside the best tool for each use-case.
It is servers as a practical starting point in terms of narrowing down the list of tools (of which there are so many), before one proceeds with a deeper dive into the best fitting tool.
Would be great if there were a site that did this sort of thing for all the common architectural needs.
(disclaimer: I work for Temporal on the Go SDK and upcoming Python SDK)
See my comment about it here: https://news.ycombinator.com/item?id=28803117
tldr; a GUI for Argo