Dabbling with Dagster vs. Airflow
davidsj.substack.com
davidsj.substack.com
Prefect is generally very easy to use. Essentially, you: (a) write a Python-based flow, which defines some job to run (with subtasks), (b) turn on an orchestrator on a server somewhere, (c) turn on an agent on a server somewhere (to run the flow when instructed by the orchestrator), and (d) connect to the orchestrator, build & apply a deployment, and run it.
I find the docs a little half baked right now. One example is that cron jobs, which one would think are essential to something like Prefect, basically can't be done (as of a month ago) without touching the Prefect UI. This is extremely odd.
I also found it fairly confusing which components were supposed to be checked into source control, and which weren't. I blame this on Python deployment generally being very odd and confusing, but Prefect docs don't make it any more clear. Prefect assumes that there's an S3-like storage that both the submitting computer (my laptop) and the orchestrator (the server) can access.
Overall I find it quite handy, and probably won't switch. It feels more lightweight than say using full Docker containers, which we probably don't need right now. The UI is nicer than Airflow's, and the orchestrator & agent are much easier on resources. It feels more reproducible. I haven't tried Prefect Cloud, and we're unlikely to (security & cost are the main reasons).
Curious if there are positive experiences with any tools here from a data orchestration perspective, esp OSS?
It looks like we added cron and other schedule types to the deployment CLI just under a month ago[1].
Over the last couple of releases, we've also made it easier to pull deployments from GitHub or bake your flow code into Docker images instead of needing S3-like storage.
As with any product, there's always more to do, so I appreciate you sharing your thoughts. More than anywhere else I've worked, community feedback is a huge driver of product enhancements and feature development. Feel free to join our Slack community[2] if you'd like to share more feedback or ask questions.
[1] https://github.com/PrefectHQ/prefect/blob/main/RELEASE-NOTES...
This makes scaling Prefect so much simpler than Airflow.
They've tried everything airflow, informatica, alteryx, etc. They've even built their own custom data flow etl in python.
I often wonder if the real issues they face is more about expectations and standards such centralized logging, easy report/artifact generation, ops management, and hiring more developer oriented data engineers.
> Marketers often already have access to data, at least in ‘read-only’ mode, and can download it in CSV format. They don’t really explore their options, not because they would need to learn SQL (no need with spreadsheets), but because this would require them to study the whole data structure of the company.
A database usually contains dozens or hundreds of data tables, and understanding how they are organized, how they relate to each other and how often they are updated is a huge effort. Therefore, they say they want to be more data-driven, but they rarely acquire the knowledge that is required to do so, because the bar is pretty high and because this would add up to their existing workload
The inverse observation here I believe is also relevant and rings true. Engineers often don't have have the marketing/sales/analytics/business domain context to make decisions that inform optimal infrastructure decisions. There's a bidirectional dog-piling effect that just gets worse over time.
My own personal experience has led me to believe that this is largely an ownership and governance issue. It's not enough to just give marketers access to the data warehouse, and it's also not enough to sync up with engineers about KPIs and OKRs on a weekly basis. The result of that is the equivalent of a kids soccer game where everyone is chasing the ball rather than playing their positions as a team.
Adding tooling complexity on top of all that, pressed into the service of fixing "pipeline woes", compounds the problem substantially.
[1] https://www.getlago.com/blog/why-we-moved-away-from-the-no-c...
* Both provide fully-managed or hybrid SAAS options. Dagster has Dagster Cloud, while Astronomer has Astro.
* Both are containerized deployments.
* Both have fully-functional local development environments. My Airflow development environment is my local env managed by the Astro CLI. It works great. I haven't worked with Dagster, but I've heard that it's lighter weight and local development is delightful.
* Both have functionality for data lineage. Dagster has Software Defined Assets, while Airflow/Astronomer integrates with OpenLineage. I don't have direct experience here, btw.
* Both are reasonably priced. They're cheap enough that the build vs buy decision is a no-brainer: buy buy buy.
One difference, however, is that Dagster natively isolates tasks in the DAG into separate Kubernetes pods. You can get this with Airflow if you use KubernetesExecutor, but KubernetesExecutor is only available in Astronomer's Hybrid-SAAS product. Only CeleryExecutor is available in Astro.
There's also a difference in syntax for defining DAGs. I find Dagster's approach is more Pythonic than Airflow's standard way of defining DAGs. However, Airflow's TaskFlow API is almost at parity with Dagster's approach.
One advantage that Airflow has over Dagster is its maturity and pre-built integrations with other systems. If I need to interact with Fivetran in Airflow, there's a provider for that. Hightouch? there's a provider. Snowflake? Yep. If there's a widely used product or managed service related to data, there's probably an Airflow provider for it.
Astronomer is extra trivial, but it's not FOSS.
Also, as other's have mentioned the comparison to MWAA is unfair, the true own of Airflow is Astronomer as they have over 50% of the commits to Airflow, and Astronomer is a much better product the MWAA.
A "quickly-implemented" feature != parity in utility. E.g. Airflow's TaskFlow superficially looks like some of Dagster's APIs, but the experience of using them is way different:
- TaskFlow is built on top of XCom, which isn't designed for data sizes that larger than small.
- If you use TaskFlow instead of the recommended practice of using Kubernetes Operators (https://medium.com/bluecore-engineering/were-all-using-airfl...), you lose the ability to isolate Python dependencies between DAGs.
Having worked on Hadoop in the past, I saw how it's easy for large Apache projects to add tons of new features to get "parity" with upstart competitors, but still lose to software like Snowflake and Spark that engineer those features in the "right" way as part of a streamlined product vision.
Much of what makes Spark successful was there at the beginning - e.g. a clean programming model and an architecture that didn't require provisioning new resources from the resource manager every time it launches a task.
On one hand, it is immensely successful Apache project.
On the other hand, it is a failure of open source - Databricks Spark is full of proprietary extensions that are not in Apache Spark. AWS Spark (in EMR and Glue) is full of proprietary extensions that are not in Apache Spark. Same with Cloudera, IBM and other players.
There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
Well, can't disagree with that.
>There's also the story how Databricks people blocked IBM patch to Apache Spark that was supposed to make it faster on IBM POWER platform - they didn't want to help competition.
Any link to that? I can't find the story.
I can't find it either, I'm pretty sure I heard it on some conference, so consider it made up. It was years ago anyway.
1. Dagster and Prefect won't ever have a hosted version on AWS/GCP (teams can spin these up from their consoles without even considering a vendor conversation or leaving their infra).
2. There are very large Airflow projects out there with huge DE teams running them, that just can't really move to Dagster or Prefect very easily. Not do they really want to if they can do what they need to in a tool they are familiar with.
My perspectives have been very much coming at the tools with fresh eyes.
Will they overtake Airflow in terms of features? They probably already have.
1.) data pipelines / data-science
2.) cicd / build pipelines
3.) ... you name it
I mean, just a service that gives me all the groundwork to build one of the above myself.
Is there something like that?
It gives you a pretty flexible job system, identities (secrets), monitoring, notifications, a web interface, a command line tool and a HTTP API. You can build lots of things above it.
Do you have specific use cases in mind ?
If you want to go even lower level, a framework like DTFx lets you define long-running, distributed and resilient orchestrations in code:
In particular, workflows are extremely durable, they can easily just sit indefinitely waiting on a condition and you can redeploy workers whenever, and workflows will resume exactly where they were.
Dabbling with Dagster vs. Angling with Airflow?
I like the fact that the articles the author has are walking the reader through their own journey, along with the learnings and opinions after the trial periods. I am curious to hear how Dagster will evolve for their usage.
One of the things that has been a big discussion point among data folks that I have been talking to is that people that haven't done ops before underestimate the amount of operations that go into managing an Airflow instance, there are still quite a lot of stuff to be figured out: - how do I get my pipeline code there? - where do I execute them? - how do I setup my development environment? - how do I make sure the platform is up and running? - how do I know if a task fails? - where do I store my logs? - how do I scale my setup?
If you are using a managed solution like MWAA or Cloud Composer, some of these questions might go away, but some aren't. As it stands today, Airflow is a powerful but hard-to-use technology; in my opinion, it is less of a tool that is supposed to be used directly by data engineers / analysts, and more of a platform that should enable easier-to-use platforms for its internal users.
In that sense, I believe Dagster is hitting the right chord: they focus on the pain points in DX for Airflow and similar solutions, they have figured out how to do development branches, they are focusing on assets rather than tasks, and they are constantly improving their product as far as I can tell from the outside. However, it is still a platform that you have to have engineers writing code for it: asking a data analyst to write python code to schedule a few SQL queries still adds a huge barrier to entry.
I am excited to see all the innovation that is happening in this space. I find the point about people getting used to the baggage around Airflow to be a quite real problem, and I am very happy to see solutions like Dagster are gaining speed. All in all, it is a very large space and there are many different problems that need to be solved to add the data abilities larger organizations have to smaller companies with limited budgets.
If you can solve all of these problems, you deserve to be rich. If you are a little unsure, perhaps take one of the segments you mentioned and focus on that.
ive looked at great expectations and thought about a uniquebuiltin solution