Flyte: A Cloud Native Machine Learning and Data Processing Platform

Great question, I am working on a follow up blog that will explain the differences in more detail. Flyte does take some inspiration from airflow, but it has a lot of important differences - Flyte natively understands data flow between tasks. This is achieved using its own type system created in protobuf - Flyte tasks are first class citizens and hence can be shared, reused and are always associated with an interface declaration - Flyte is container and kibernetes native. It is also multi tenant. - Flyte corn scheduler, control plane api and the actual execution engine are decoupled. Each workflow can be independently executed on a different execution engine - Flyte workflows are purely specification - defined in protobuf and Flyte tasks also - Flyte provides an event stream of the execution - since Flyte is aware of the data, it comes with built in memorization and auto cataloging - like airflow Flyte can have plugins in python, but it supports a richer plugin interface - Flyte is written in Golang and on top of kuberenetes It is definitely less mature in the open source, so please help us make it better. But it has been battle tested at Lyft for more than 3 years in production.

Here is a blog post I wrote a few weeks ago that describes how Flyte's structured workflow specification allows for using open-source workflows with no code.

https://medium.com/@flytehub/introducing-flytehub-open-sourc...

Airflow just cover authoring, monitoring and scheduling. I don't recall it covered caching, lineage nor resource allocation/reuse.

It definitely feels like there is Airflow inspiration though poking through the site and docs, it seems that the devil is in the details with respect to the differences between the two... kinda like the relationship between Airflow and Prefect [1]. It looks like the barrier to entry is higher with Flyte but that there are benefits as well.

One example I see is that one can run Airflow simply without containers if desired with just simple Python functions whereas Flyte seems to be much more concerned with managing the execution environment for you (pros/cons to that).

Flyte also seems to be more "Kubernetes native" by default [2][3] vs with Airflow this is more of a choice amongst several executors.

I'd be curious to see a performance benchmark using comparable workflows vs Airflow with the Kubernetes Executor or Kubernetes Operator.

[1]: https://www.prefect.io/

[2]: https://kccncna19.sched.com/event/UaYY/flyte-cloud-native-ma...

[3]: https://www.youtube.com/watch?v=KdUJGSP1h9U

That is something to which the SDK was designed to be extended--in fact, the SDK was designed specifically to be extended to different programming models besides the rudimentary one we provide out-of-box. However, we have not had the time to take on the implementation ourselves as the existing one serves our purposes at Lyft well and we are but a small team. So we do hope that someone in the community with a knack for syntax can help improve things!

You mean DataClasses?

It is definitely possible to leverage Flyte's features here for sharing partial outputs. It does, however, require formatting your pipelines in a particular manner for this to be supported natively (i.e. create a task which computes a view or similar. Then that task is shared among different pipelines). It might be a little verbose, but, in my opinion, it is preferable because it modularizes pipelines into tasks which are individually tractable for testing and validation--especially as the tasks and pipelines evolve in a large organization! Further, the verbosity of authoring such pipelines can be reduced significantly by making good use of the flytekit library.

Additionally, there is always the option of introducing a custom plugin. Although it would take more effort up front, one can really let their imagination run wild and introduce behaviors as needed.