* Conductor implements a workflow orchestration system which seems at the highest level to be similar to Airflow, with a couple of significant details.
* There are no "workers", instead tasks are executed by existing microservices.
* The Orchestrator doesn't push work to workers (e.g. Airflow triggering Operators to execute a DAG), instead the clients poll the orchestrator for tasks and execute when they find them.
My hot take:
If you already have a very large mesh of collaborating microservices and want to extract an orchestration layer on top of their tasks, this system could be a good fit.
Most of what you're doing here can also be implemented in Airflow, using an HTTPOperator or GRPCOperator that triggers your services to initiate their task. You don't get things like pausing though. On the other hand, you do get the ability to run simple/one-off tasks in an Airflow operator, instead of having to build a service to run your simple Python function.
I'm unsure on whether push/pull is better; I think it largely depends on your context. I'm inclined to say that for most cases, having the orchestrator push tasks out over HTTP is a better default, since you can simply load-balance those requests and horizontally scale your worker pool, and it's easier to test a pipeline manually (e.g. for development environments) if the workers respond to simple HTTP requests, instead of having to provide a stub/test implementation of the orchestrator. (In particular I'm thinking about "running the prod env on your local machine in k8s" -- this isn't practical at Netflix scale though.)