Hacker News Reader

Comment by nojito | Hacker News Reader

Airflow is a great example of technology being used at a massive company for massive company problems...which is now being pushed as a solution to everything

Papermill is another example.

View on HN

When Max started building airflow Airbnb's engineering team was probably 80 or so people. The data engineering and data infrastructure teams couldn't have been more than ten combined. I'm not sure that's massive but it is the point where the previous solution (Chronos: https://medium.com/airbnb-engineering/chronos-a-replacement-...) was starting to strain.

Airflow may not be necessary for a ten person team but it could be if that team has complicated data orchestration needs and doesn't want to incrementally replace their infrastructure every couple years.

Airflow can be used to process data that only adds up to 10s or 100s of MBs. In aggregate, a big tech company processes petabytes or exabytes, but remember, technology like Airflow isn't processing all of that simultaneously within one task. So no, I would not characterize Airflow as a solution to just "massive company problems".

Been using airflow in Production at a small start-up for almost 4 years now and been very happy with it (small dockerized deployment, and simple syncing with github) - what big company problems does it solve that make it unsuitable for use at small companies?

Any hints at solutions that help if you're a small company having small company sized problems but don't want to DIY the entire flow execution logic?

I'm the lone developer on a project which will likely never scale beyond a few thousand users and I'm really liking Dagster. You can deploy it on a bunch of other platforms (Airflow, Dask, Celery, K8s) which is really nice for my use case (automating workflows in HPC environments from the browser) or run it standalone

We were a very small company when we discovered Airflow, just saying, and it provided a lot of value to us vs. alternatives.

We fell in love with Airflow's capability to allow us to write one small python program that generated hundreds of tailored dags by iterating over the configuration of each of our customers. This dynamism is very powerful for similar use cases.

I have set it up, configured and performed all the upgrades just by myself in a small startup, performing very complex daily tasks.

I miss features, sure. The whole execution timing might be confusing the first time, sure. But I can't figure out why i see so many comments regarding deployment difficulties.

Reply on news.ycombinator.com