Hacker News Reader

Comment by Hasz | Hacker News Reader

Dated doesn’t mean bad (usually the opposite in my experience!) What issues do you have with Airflow?

Here's my problems with MWAA (amazon hosted airflow.) I have about 100 dags which maxes out the scheduler thread. Airflow parses all the files every minute so it's always parsing around 94% cpu. I could run a second scheduler thread if I coordinate with my SRE team and get the terraform deployed...it's really tedious.

Related possibly, my dags get kill -9 for no apparent reason. The RAM usage is not that high, maybe 2gb out of 8gb system RAM in use. No reason is given in the logs.

I am trying to switch to dagster, not because it's awesome, but because it hasn't crashed randomly on me.

MWAA is hot garbage. I had similar issues and switched to running it on EKS instead.

This feels like an MWAA issue but I understand how that often gets conflated with it being an Airflow issue.

You're right, it doesn't happen when developing locally, only in MWAA. This was the answer given by the Airflow team as well and I figured they would punt before I asked.

I realize Amazon is taking an open source project and making a ton of money on it (the instance prices are ridiculous for what you get) and the incentives are misaligned for the Airflow team to help AWS make it better unless AWS paid them to help fix it.

It's crap all around, and Airflow gets a bad rap from AWS's terrible MWAA product based on it.

> What issues do you have with Airflow?

Their operational perspective is catastrophic; how does one view the logs for a dag through the UI[1]? Why can't it store the python in the database they have attached to their deployment, versus making me jump through 80,000 hoops to put the files in the right magic directory on disk of every worker[2]?

1: no, not <https://airflow.apache.org/docs/apache-airflow/stable/ui.htm...> I mean the log, you know, like in the old days of $(tail -f /var/log/the.thing). I'm open to the answer hiding somewhere in this gobbledygook <https://airflow.apache.org/docs/apache-airflow/stable/admini...> but who is the target audience for having such a fancy UI and omitting log viewing from it, doubly so if there's some alleged http just for viewing logs

2: https://airflow.apache.org/docs/apache-airflow/stable/core-c... and double-plus-good anytime python software mentions PYTHONPATH -- that's how you know you're in for a hot good time https://airflow.apache.org/docs/apache-airflow/stable/admini...

We deploy on K8s in OpenStack from a scheduled GitHub Actions pipeline which aggregates DAGs into a new container build based on hashes of hashes. This works well with almost no intervention.

WRT your 1, above any DAG output to stdout/err is available via the logs tab from the graph view of the individual tasks. Almost all our DAGs leverage on the PythonOperator though, not sure if that standardises this for us and your experience is muddied by more complexity than we currently have?

WRT 2. we generate an uber requirements.txt running pyreqs from the pipeline and install everything in the container automatically. Again no issues currently - although we do need to manually add the installation of test libraries to the pipeline job as for some reason auto-discovery is flakier for unit-tests frameworks.

Reply on news.ycombinator.com