I’m curious as to if anyone can say how this compares to dagster since both libraries seems to rely on deploying to engines like Airflow?
- pipeline syntax is absolutely minimal (even supporting lambdas for simple transitions), inspired by the Clojure library core.graph https://github.com/plumatic/plumbing
- sequential and parallel runners are built-in (don't have to rely on Airflow)
- io provides wrappers for existing familiar data sources, but directly borrows arguments from Pandas, Spark APIs so no new API to learn
- flexibility in the sense you could rip out anything, for example, the whole Data Catalog replacing with another mechanism for data access like Haxl
- there's a project template which serves as a framework with built-in conventions from 50+ analytics engagements