At risk of running out my question quota: we use dagster at the moment but have found that it has some unexpected rough edges. Would you be able to point me in the direction of your preferred comparison against Prefect?
I think one of the key differences between Prefect and Dagster is that Prefect views orchestration as coordination, while Dagster views orchestration as reconciliation. The data stack is a complex system whose state is frequently mutated by forces outside our users’ control. Therefore, our product is focused on letting our users understand and react to those events, no matter where they come from. That could include everything from scheduling fully-orchestrated Prefect pipelines, to setting an SLA for database maintenance that Prefect doesn’t have anything else to do with. Reconciliation, in contrast, requires users to define a digital twin of their stack, in order to serve as ground truth and become the reconciliation target. Philosophically, we view Prefect as one piece of an ever-changing stack. We are focused on being as flexible as possible to fit into the stack, rather than the other way around.