Hacker News Reader

Comment by 0x500x79 | Hacker News Reader

Some people have noted that this is a very Airbyte specific article, but I think that the lessons learned are still important.

I have managed Airflow as a managed service for a company that has thousands of DAGs and one of our keys to success was splitting the compute and scheduling concepts into different components. We standardized on where our compute ran (Databricks, Spark, Lambdas, or K8s jobs) and had Airflow purely as a scheduler/orchestration tool.

Scaling your Airflow worker nodes to handle big-data scale transformations/extractions is a pain. Especially attempting to support customers who want to run larger and larger jobs. Splitting these concepts allowed for us to prevent noisy neighbor issues, Airflow as a component had high reliability for all of our customers, and we prevented the need for M * N operators.

View on HN

In my org we never use Airflow to compute anything. We only use a single operator: PythonOperator.

All business and data access logic is encapsuled in REST APIs. Our DAGs are responsible for calling those APIs in the correct order, in the correct time, retrying when needed. I really dislike using Airflow for anything else, as it generally becomes a huge mess.

If it's all Python, why not use Prefect?

First time I read about Prefect, can you briefly share why you would use prefect over airflow if using only python? (that's what I am about to be doing soon...)

Have been a Prefect user since their first release. In a nutshell, it's a native pythonic DAG tool, where Airflow is a DAG tool that happens to be written in Python.

It didn't exist when we bootstrapped our department. Nowadays Airflow is a dev favorite and we have no good reason to seek alternatives.

How best to achieve this? I'm considering Astronomer or AWS hosted airflow, with a benefit to AWS having the compute components easily accessible within the AWS ecosystem.

Starting out with a smaller scale and lower commitment to Airflow, so I'd like highest reliability with least hassle.

https://aws.amazon.com/blogs/aws/introducing-amazon-managed-...

At Grove we make extensive use of KubernetesPodOperator[0]. This allows to both encapsulate the job in containers (k8s pods), and also specify the resources we need for each DAG.

[0] https://airflow.apache.org/docs/apache-airflow-providers-cnc...

It depends on what other tools you are currently using for ETL (or want to use). One example is that we used Spark so we would use the Spark submit operator to submit jobs to clusters. You can also use the K8s Pod operator if you want to utilize containers for your compute.

There are a lot of options. We were adopters before AWS hosted airflow was a thing, so I don't have any experiencing running AWS hosted Airflow.

I haven't looked recently to see if some of the challenges we faced early on are solved now, but most of them stemmed from how DAG updates were handled: changing the start date on a DAG would break you DAG forever until you go update the database by hand. Things like this are(/were?) super painful and could get worse with a managed solution.

Thanks. Requirement at this stage is to coordinate the running of a few singer.io or airbyte taps.

I may consider Prefect over Airflow, generally my experience so far resonates with this blog post: https://towardsdatascience.com/tired-of-airflow-try-this-c51...

If you are on aws, then step functions could be your best option. They are responsible for the workflow definition, and the compute load is completely dependent on the underlying tasks and aws services you are going to use.

Edit: typos

Amazon's Airflow offering sucks really hard. As in many services, they made an awful job at designing a good UX and hiding the product's complexity from their users.

Astronomer is cool, but expensive and they won't accept monthly billing. Also, it's even more expensive if you need DAGS to access things within a VPC.

Step Functions is decent for simple use cases and very reliable. For complex stuff, you will hate their JSON based DSL with passion.

Thanks, glad to hear that before I bother trying it.

Have heard astronomer is too expensive.

Step Functions doesn't look great either.

We used Astronomer at my last job. Sure beats trying to run your own instance and the Astronomer team were fantastic.

Reply on news.ycombinator.com