vtuulos

If you are ok with executing your SFN steps on AWS Batch, Metaflow should do the job well. It's pretty inhuman to interact with SFN directly.

One feature that's in our roadmap is the ability to define DAG fully programmatically, maybe through configs, so you will be able to have a custom representation -> SFN JSON, just using Metaflow as a compiler

Stay tuned! We have some cool new features coming soon to support agentic workloads (teaser: https://github.com/Netflix/metaflow/pull/2473)

If you are curious, join the Metaflow Slack at http://slack.outerbounds.co and start a thread on #ask-metaflow

Metaflow tracks all artifacts and allows you to build dashboards with them, so there’s no need to use MLFlow per se. There’s a Metaflow integration in Weights and Biases, CometML etc, if you want pretty off-the-shelf dashboards

Metaflow was started to address the needs of ML/AI projects whereas Airflow and Dagster started in data engineering.

Consequently, a major part of Metaflow focuses on facilitating easy and efficient access to (large scale) compute - including dependency management - and local experimentation, which is out of scope for Airflow and Dagster.

Metaflow has basic support for dbt and companies use it increasingly to power data engineering as AI is eating the world, but if you just need an orchestrator for ETL pipelines, Dagster is a great choice

If you are curious to hear how companies navigate the question of Airflow vs Metaflow, see e.g this recent talk by Flexport https://youtu.be/e92eXfvaxU0

I don't know if it's a coincidence but we just released a major new feature in Metaflow a few days ago - composing flows with custom decorators: https://docs.metaflow.org/metaflow/composing-flows/introduct...

A big deal is that they get packaged automatically for remote execution. And you can attach them on the command line without touching code, which makes it easy to build pipelines with pluggable functionality - think e.g. switching an LLM provider on the fly.

If you haven't looked into Metaflow recently, configuration management is another big feature that was contributed by the team at Netflix: https://netflixtechblog.com/introducing-configurable-metaflo...

Many folks love the new native support for uv too: https://docs.metaflow.org/scaling/dependencies/uv

I'm happy to answer any questions here

if you want to see similar tricks applied in Python (with a JIT compiler for query-time optimization), take a look at this fun deck that I presented a long time ago: https://tuulos.github.io/sf-python-meetup-sep-2013

we were able to handle trillion+ datapoints with relatively modest machines - definitely a useful approach if you are ready to do some bit twiddling

yes, this. In case you are interested in seeing some numbers backing this claim, see here https://outerbounds.com/blog/metaflow-fast-data

Source: I used to work at Netflix, building systems that pull TBs from S3 hourly

I had the same concern. However, the structure of the output was surprisingly stable. We rejected badly formatted responses: https://github.com/outerbounds/hacker-news-sentiment/blob/ma...

The semantics of the topics/tags could be improved for sure with a more detailed prompt

here's how the model ranks the discussion on this page after 40 comments:

SENTIMENT 6

:D

yep! And of course the new 3.1 model

even simpler, you can just do it in SQL

You can find all titles and dates since the beginning of HN in this public BigQuery dataset: https://console.cloud.google.com/marketplace/product/y-combi...

That's an interesting hypothesis but the words we use to express agreement and disagreement haven't changed much.

We don't try to retrieve articles/topics from the model, which would be affected by the cutoff, just asking it to analyze the sentiment or summarize the content provided in a prompt

yes please! The data is conveniently available as JSON blobs here https://github.com/outerbounds/hacker-news-sentiment/tree/ma...

search "divisive" here: https://github.com/outerbounds/hacker-news-sentiment/blob/ma...

I actually spent 10 minutes trying to see if there are obvious tests for U-shaped distributions. I'd love to hear if anyone has ideas here.

Exactly. Now we have a new UI paradigm and a new, incredibly powerful NLP tool in our toolkit.

Is this AGI? Of course not. Is this useful and valuable? Obviously.

here's how we solved engineering challenges related to RAG using open-source Metaflow: https://outerbounds.com/blog/retrieval-augmented-generation/

Metaflow integrates with AWS Batch which many folks use for serious HPC. Internode scheduling happens through the multinode scheduling supported by AWS Batch. networking via EFA etc.

We'll blog more about this soon but you can certainly give it a try today! https://github.com/outerbounds/metaflow-ray

Congrats for the launch Stefan and Elijah! :)

Like Stefan mention in the OP, Hamilton works well with tools like Metaflow which can help with many other concerns you mentioned. How you define your data transformations for ML is an open question that Hamilton addresses neatly.

See here for an example of Metaflow+Hamilton in action: https://outerbounds.com/blog/developing-scalable-feature-eng...

Outerbounds | Systems / Full-Stack engineer | SF / Remote | Full-time

Outerbounds commercializes Metaflow, a widely used open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/full-stack/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, Kubernetes, or low-level systems programming.

- Outerbounds: https://outerbounds.com

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.com/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email: workwithus@outerbounds.co

Totally! One intended use case of this integration is that it allows you to move easily to a more scalable orchestrator if you hit limits of Airflow

This is great! If you are interested in running ML workloads across clouds, Netflix's Metaflow will officially announce support for all clouds tomorrow: https://outerbounds.com/blog/metaflow-on-all-major-clouds/

Quite a coincidence :)

If you want to experiment with something similar by yourself and you don't have the patience to wait for Stable Diffusion to crunch through thousands of images on your laptop or in a Colab notebook, here's how you can parallelize processing relatively easily on AWS Batch or Kubernetes: https://outerbounds.com/blog/parallelizing-stable-diffusion-...

Thanks for sharing! I am still proud of this hack :)

Outerbounds | Systems dev | SF / Remote | Full-time

Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.com/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email: workwithus@outerbounds.co

Yep, Kedro and Metaflow are more similar to each other than to other generic DAG orchestrators like Airflow.

Kedro and Metaflow make it easier to develop robust ML projects where orchestration plays an important role but it is not everything. They are two separate projects, so the way how they approach the problem differs greatly in details.

Outerbounds | Systems dev | SF / Remote | Full-time

Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.co/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email: workwithus@outerbounds.co

Outerbounds | Systems dev | SF / Remote | Full-time

Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.co/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email: workwithus@outerbounds.co

There are a few different ways to deal with spot interruptions. First, it is a good idea to specify multiple instance types in your compute environment so even if some instances types become unavailable in spot, Batch can use another type automatically.

Second, you can rely on Spot Fleets which handle both spot and on-demand instances seamlessly https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fle...

Outerbounds | Frontend dev | SF / Remote | Full-time

Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for stunning colleagues who can help us build delightfully usable, technically non-trivial UIs. This is a great greenfield opportunity for a senior frontend engineer who wants to build a product from scratch.

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.co/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email me: ville@outerbounds.co

As the article mentions, Metaflow will start supporting Kubernetes natively soon, although data scientists don't need to care about it :) Nothing changes in your Metaflow code when you move e.g. from AWS to Azure, so Metaflow isn't fundamentally dependent on AWS in any way.

Netflix is an AWS shop, so naturally we started with AWS integrations.