Hacker News Reader

Comment by simo7 | Hacker News Reader

I would add these gotchas/recommendations:

- Airflow the ETL framework is quite bad. Just use Airflow the scheduler/orchestrator: delegate the actual data transformation to external services (serverless, kubernetes etc.).

- Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).

- Don't use it for latency-sensitive jobs (this one should be obvious).

- Don't use sensors or cross-DAG dependencies.

So yeah unfortunately it's not a good fit for all the use cases, but it has the right set of features for some of the most common batch workloads.

Also python as the DAG configuration language was a very successful idea, maybe the most important contributor to Airflow success.

View on HN

> - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).

You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job.

This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it.

> - Don't use it for latency-sensitive jobs (this one should be obvious).

IIRC this is being addressed in Airflow 2.0

> - Don't use sensors or cross-DAG dependencies.

This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags.

context: I built/manage my company's Airflow platform. Everything is managed on k8s.

> You can totally design your tasks to be idempotent

Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent).

> I've never ran into issues with cross dag dependencies

I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.

> context: I built/manage my company's Airflow platform. Everything is managed on k8s

My team is running Airflow on a single node but we're slowly outgrowing this setup. We're considering running jobs on k8s.

Curious what's your setup like? Is your cluster of a fixed size or does it scale with the load?

Using the KubernetesPodOperator for everything adds a huge amount of overhead. You still need Airflow worker nodes, but they're just babysitting the K8S pods doing the real work.

I know it's 2020 and memory is cheap or whatever, but Airflow is shockingly wasteful of system resources.

re: ETL framework - you can get a lot done with the built-in Airflow Operators, including the PythonOperator (and bring in any python dependency you like) and BashOperator (call any CLI, etc.) - it's not drag-and-drop, but I've found it to be quite versitile.

re: idempotency - yes, make your workflow tasks idempotent.

re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved

re: sensors, there is some great work from Airbnb to improve: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+A...

If using a docker or kubernetes operator, you can't make use of operators or Airflow connections. Do you just work without those?

Correct, I use operators only to delegate the actual workload to an external service.

Got it, to be clear - do you still use the Airflow for storing connections, or no? If no, how do you store your credentials? We've only done a POC and we've discovered a higher than expected learning curve.

This is handy thanks - I deal mostly with luigi and this helps me place airflow a bit better.

Sensors are quite helpful when you don't know the exact moment your data will come in.

I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.

I really disagree here. Monolithic dags are a bigger pain to manage.

Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily.

We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.

What do you mean by bookmark in this context?

"I have processed 5/15 records, next run I need to start at record 6". Bookmarking is a common concept for working through a large job in several small runs.

I wonder if this difference in jargon has origins in different sub-fields. I recognize that concept as "checkpoints" across different companies but I also remember seeing the term from data science folks and thinking that I just don't know the concept.

Often called a Waterline/Watermark as well.

I'd call it a watermark too; "checkpoint" can mean an intermediate point in a multi-step database transaction.

I've always called them offsets.

I would call it "a read cursor (that's updated within the transaction)", but I'm boring.

Cursor nomenclature is used for describing where in a sequence the operator is during a code execution pass in database query or similar right?

I think I kind of immediately understand "cursor" the best in this context. I also agree, it's a little boring and definitely old school:).

Reply on news.ycombinator.com