Argo Workflows v3.0 | Hacker News Reader

Argo Workflows v3.0

blog.argoproj.io

68 pointsdnsmichi11 comments

I mainly became familiar with Argo Workflows when looking at/lightly evaluating Argo Events to run my home-(k8s)-cluster's Event Driven Architecture (as an esb/lambda-architecture/event-fabric/&c system). Also poked at Brigade and Tekton (who sadly are now CI/CD focused, not general EDA) and local Github Actions runners.

Workflow was mentioned in the Argo Events docs a couple times. Events is a pretty general system, but if you need to assemble more complex inter-dependent tasks, they had Workflow integration. I didn't get super far in, look very deep at how the integration worked, or how pleased I was going to be using Workflows, as I was mainly interested in the EDA end to start.

Anyhow, picking through Workflow 3.0, my main interest is:

> Argo Workflows v3.0 comes with a new UI that now also supports Argo Events! The UI is also more robust and reliable.

Really seems like Argo Workflow has been made the over-arching UI for both of these systems in this 3.0 release. There's now some event-flow pages in Workflow 3.0, that will be interesting to check out. I'm very interested in how the APIs have all been updated, how the systems work together. Especially given Argo Event's fall 1.0 release & the new architecture[1].

One other cool feature from the Workflow 3.0 is Widgets. Embed some code in other pages so you can see build status, job status, &c.

[1] https://blog.argoproj.io/argo-events-v1-0-released-b69668de0...

Speaking as someone whose spent the last 18 months building out Tekton pipelines, I can confidently disagree that it’s architecture (or general ethos) is built around CI/CD. The docs might allude to a lot of CI/CD use cases but the framework is very general in that you have the building blocks for any event driven workflow: Tasks, Pipelines and Events.

Tekton was very high in my runnings, very attractive. I liked the models they built around, they seemed like a good structure. Everything built very "cloud-native" (system state stored as Kubernetes objects).

But the project did explicitly pivot, sometime in the last 12 months, to explicitly declare themselves/think of themselves as a CI/CD system. They're also under the Continuous Delivery Foundation now. I spent 5 minutes looking for the specific event, where this was determined/declared, & will try to follow up; hope I find it.

I don't think it precludes Tekton from still being used more generally, thought of more generally. But now, doing so puts one out of alignment with the stated goals, with the project itself. I'm quite sad about this development: I think it radically undershoots the relevance of Tekton & what role it ought to fill. We have all this wonderful new cloud based pieces of state, but in terms of how one piece actuates another, how we can make systems that behave autonomically: we direly need some core Event Driven Architecture logic, and Tekton definitely has some really good tools for doing that. I want very much for Tekton to meet it's potential head on.

Why would I use this over airflow?

If you prefer yaml > Python or if you prefer installing one k8s app instead of managing all of the airflow dependencies (scheduler, webserver, workers, etc)

Here's a conference talk that gives a detailed side-by-side comparison: https://youtu.be/oXPgX7G_eow

I'm strongly considering moving my fairly immature Airflow pipeline to Argo Workflows because:

* the Airflow DAG deploy/versioning is surprisingly primitive. The best option here seems to be to use the KubernetesOperator to version your steps, and if you're using k8s to execute, why not use it for the rest?

* the Airflow UI is pretty confusing to use, maybe this gets easier once you know your way around it.

* my team has k8s expertise and we don't know Airflow well yet; seems like less to learn running Argo Workflows, assuming you're already fluent in k8s.

* if you're already running k8s, it seems like you have to add fewer components to get Argo running; more duplication with Airflow-on-k8s.

On the other hand, being able to unit test / locally run your DAGs on your dev machine is a big plus for Airflow, where Argo Workflows seem to have a less strong testing story. And writing YAML is not preferable to writing Python DAG files.

Hi @theptip,

I'm an Airflow PMC and would love to know a bit more about your comparison :).

1. Have you tried Airflow 2.0? We made some pretty big overhauls both in terms of UI and backend. 2. DAG versioning is currently problematic, but DAG versioning is a "when" and not an "if" so should be in a future 2.x version :). That said could you describe a bit more about your deployment issues? User stories like this help us improve the product. 3. Have you looked into using KEDA with the CeleryExecutor? You could create KEDA queues for a lot of commonly used workflows and then you'd only need to use the python or bash operator to run those tasks instead of k8spodop. 4. Are you using the Airflow helm chart or did you custom roll a deployment?

Any feedback would be highly appreciated and I'm also glad to answer any questions you might have!

Thanks for getting in touch, happy to share.

1. We ended up using GCP's hosted Composer to get started more quickly, which doesn't seem to have been updated to Airflow 2.0 yet. I'll put that on the list for evaluation.

2. A few usecases that I immediately hit complexity walls on:

A) Having a "staging" version of our pipelines so that we don't break the prod ETL; it was really difficult to find a canonical method for having common DAG code that's parameterizable per env. The fact that all of the DAGs live side-by-side in the same directory means I have to run the same job for a "prod push" as a "staging push" (i.e. if I get the staging deploy wrong I could break prod). Given that we deploy version vN+1 to staging, check it's working, and only then deploy vN+1 to prod, we ended up with some weird config injection code to let us have two folders containing copies of the same DAG scripts with different config. This just felt janky.

B) Managing Python dependencies between different apps was also painful; for example we wanted to add Meltano, and so that app brings in a bunch of deps, which broke our main dags when I naively updated the main python pip env to install the new meltano requirement. Using the K8s operator lets us effectively have a venv per dag but the pattern of using one python env across the whole Airflow install bit me very early on and seemed pretty unscalable.

3. I haven't looked at KEDA, I'll take a look.

4. We're using GCP Composer for now, though I looked at the Helm chart too.

There's actually a python dsl that compiles to YAML: https://github.com/argoproj-labs/argo-python-dsl

and I'm sitting where wondering, why would you use Airflow over Argo Workflows?

Everyone should evaluate the options for their own needs.

DAG or looping/near-Turing? If DAG only, can it subflow/recurse, and to what depth?

What is the underlying datastore that enables scaling?

Primary integrations?

How are custom integrations done? "Web service call"? Spinnaker kinda sucks that way.

Stop/Resume/Edit state to correct errors?

Suitability for builds? (e.g. jenkins)

Suitability for deploys? (e.g. spinnaker)

Reply on news.ycombinator.com