Metaflow: Build, Manage and Deploy AI/ML Systems
github.com
github.com
A big deal is that they get packaged automatically for remote execution. And you can attach them on the command line without touching code, which makes it easy to build pipelines with pluggable functionality - think e.g. switching an LLM provider on the fly.
If you haven't looked into Metaflow recently, configuration management is another big feature that was contributed by the team at Netflix: https://netflixtechblog.com/introducing-configurable-metaflo...
Many folks love the new native support for uv too: https://docs.metaflow.org/scaling/dependencies/uv
I'm happy to answer any questions here
Straightforward for data/ML scientists to pick up, familiar python class API for defining DAGs, and simplifies scaling out parallel jobs on AWS Batch (or k8s). The UI is pretty nice. Been happy to see the active development on it too.
Currently using it at our small biotech startup to run thousands of protein engineering computations (including models like RFDiffusion, ProteinMPNN, boltz, AlphaFold, ESM, etc.).
Data engineering focused DAG tools like Airflow are awkward for doing these kinds of ML computations, where we don't need the complexity of schedules, etc. Metaflow, imho, is also a step up from orchestration tools that were born out of bioinformatics groups, like Snakemake or Nextflow.
Just a satisfied customer of Metaflow here. thx
Most of the time this not a blocking problem since each step in a flow is mapped to a Docker image and/or your choice of EC2 instance (e.g. one step on a GPU, another on a memory optimized instance). You can have one step use an image with all of your python-based ML stuff, and another step have a different image with compiled exectuables that are triggered by a system call. If needed, outputs from such a system call would then need to be persisted in a database/S3 or read back into the python flow for persistence. So, it is not as seamless as a flow in all python, but it can work "good enough".
If you squint a bit it's sort of like an Airflow that can run on AWS Step Functions.
Step Functions sort of gives you fully serverless orchestration, which feels like a thing that should exist. But the process for authoring them is very cumbersome - they are crying out for a nice language level library i.e. for Python something that creates steps via decorator syntax.
And it looks like Metaflow basically provides that (as well as for other backends).
The main thing holding me back is lack of ecosystem. A big chunk of what I want to run on an orchestrator are things like dbt and dlt jobs, both of which have strong integrations for both Airflow and Dagster. Whereas Metaflow feels like not really on the radar, not widely used.
Possibly I have got the wrong end of the stick a bit because Metaflow also provides an Airflow backend, which I sort of wonder in that case why bother with Metaflow?
Consequently, a major part of Metaflow focuses on facilitating easy and efficient access to (large scale) compute - including dependency management - and local experimentation, which is out of scope for Airflow and Dagster.
Metaflow has basic support for dbt and companies use it increasingly to power data engineering as AI is eating the world, but if you just need an orchestrator for ETL pipelines, Dagster is a great choice
If you are curious to hear how companies navigate the question of Airflow vs Metaflow, see e.g this recent talk by Flexport https://youtu.be/e92eXfvaxU0
Here's a nice article with code examples implementing a simple pipeline: https://www.quantisan.com/orchestrating-pizza-making-a-tutor....
Having said that, they have slightly improved the StepFunctions by adopting JSONata syntax.
You just want some Python code that builds up a representation of the state machine, e.g. via decorating functions the same way that Celery, Dask, Airflow, Dagster et al have done for years.
Then you have some other command to take that representation and generate the actual Step Functions JSON from it (and then deploy it etc).
But the missing piece is that those other tools also explicitly give you a Python execution environment, so the function you're decorating is usually the 'task' function you want to run remotely.
Whereas Step Functions doesn't provide compute itself, it mostly just gives you a way to execute AWS API calls. But the non control flow tasks in my Step Functions end up mostly being Lambda invoke steps to run my Python code.
I'm currently authoring Step Functions via CDK. It is clunky AF.
What it needs is some moderately opinionated layer on top.
Someone at AWS did have a bit of an attempt here: https://aws-step-functions-data-science-sdk.readthedocs.io/e... but I'd really like to see something that goes further and smooths away a lot of the finickety JSON input arg/response wrangling. Also the local testing story (for Step Functions generally) is pretty meh.
One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler
Seems like something new to learn, an added layer on top of existing workflows, with no obvious benefit.
My opinion about Netflix OSS has been pretty low as well.
I find google is purpose built for ml and provides tons of resources with excellent documentation.
AWS feels like driving a double decker bus, very big and clunky, compared to google, which is a luxury sedan, that is quite comfortable to take you where you’re going.
It may look redundant on the surface, but those cloud services are infrastructure primitives (compute, storage, orchestration). Metaflow sits one layer higher, giving you a data/model centric API that orchestrates and versions the entire workflow (code, data, environment, and lineage) while delegating the low‑level plumbing to whatever cloud provider you choose. That higher‑level abstraction is what lets the same Python flow run untouched on a laptop today and a K8s GPU cluster tomorrow.
> Adds an extra layer to learn
I would argue that it removes layers: you write plain Python functions, tag them as steps, and Metaflow handles scheduling, data movement, retry logic, versioning, and caching. You no longer glue together five different SDKs (batch + orchestration + storage + secrets + lineage).
> lacks concrete examples for implementation flows
there are examples in the tutorials: https://docs.outerbounds.com/intro-tutorial-season-3-overvie...
> with no obvious benefit
There are benefits, but perhaps they're not immediately obvious:
1) Separation of what vs. how: declare the workflow once; toggle @resources(cpu=4,gpu=1) to move from dev to a GPU cluster—no YAML rewrites.
2) Reproducibility & lineage: every run immutably stores code, data hashes, and parameters so you can reproduce any past model or report with flow resume --run-id.
3) Built‑in data artifacts: pass or version GB‑scale objects between steps without manually wiring S3 paths or serialization logic.
But have not seen anyone talk about it in that context. What do people use for AI workflow orchestration (aside from langchain)?
If you are curious, join the Metaflow Slack at http://slack.outerbounds.co and start a thread on #ask-metaflow
They call it a DREAM stack (Daft, Ray Engine or Ray and Poetry, Argo and Metaflow)