Hacker News Reader

Comment by dlphn___xyz | Hacker News Reader

how does this compare to Luigi or Airflow?

I would strongly suggest not using Airflow if your company doesn't already... and that's coming from someone working at the company that made it.

1) It has no knowledge of data dependencies. This means task success is very primitive and you end up with a bunch of "polling" tasks waiting eg; for a table partition to land. 2) The UI constantly hangs and/or crashes 3) Airflow "workers" using Celery are rarely correctly given the right numbers of tasks. OOM-ing, etc. are all commonplace even if using Docker.

And many, many more.

I strongly suggest using Apache Beam or Argo w/ Kubernetes instead. They can scale quite a bit more and deal with long running tasks well.

I'm actually finding airflow + beam + kubernetes to be a really powerful combination. Especially if you're on GCP where airflow is managed by Cloud Composer, kubernetes is managed by GKE, and beam is managed by Dataflow.

Our airflow cluster _does_ almost nothing. Our operators just tell kubernetes to run containers with commands or dataflow to run some beam template and waits for the results from afar.

I love this setup but man do I agree with your points about the UI. For a product so powerful I can't believe how problematic it is.

The biggest difference besides the programming model is scale. Luigi and Airflow target data pipelines that don't require much scale. Cadence is built to support business level transactions. It can handle tens of thousands of events per second and hundreds of millions of open workflows. Obviously it is also a good fit for low scale use cases. Cadence is also so generic that it can be used to implement practically any workflow definition language. For example it is possible to create an extension to run Airflow pipelines on Cadence.

I'm from the Cadence team.

> Obviously it is also a good fit for low scale use cases.

Is this a given? Just because something is necessary at scale doesn't mean it is a good fit for low scale use cases. I would expect the opposite is actually true.

We have both very low scale use cases as a single distributed cron as well as very high scale use cases in production at Uber.

As someone who uses (but doesn't love, actually) Airflow. I think this might be a situation where lots of people are tempted to think that they're "scale". I disagree slightly with your use of "business level" to describe this.

I've seen quant hedge fund platforms(fairly intensive data engines compared to most businesses) using airflow, because when it comes down to it hundreds of DAGs is more likely than hundreds of thousands.

Cadence says it's meant for long-running tasks, while Luigi tasks are supposed to be short.

Cadence supports both long-runnig as well as very short tasks.

I didn't think that Cadence would fail if the task finished too quickly. The point is that Cadence is designed to solve a slightly different problem than Luigi.

Reply on news.ycombinator.com