savin-goyal

the topic of managing large dependency chains for ML/AI workloads in a reproducible has been a deep rabbit hole for us. if you are curious, here is some of the work in open domain

https://docs.metaflow.org/scaling/dependencies https://outerbounds.com/blog/containerize-with-fast-bakery

what's up with the title flips from

> 350M Tokens Don't Lie: Love And Hate In Hacker News, to

> LLM-based sentiment analysis of Hacker News posts, to

> LLM-based sentiment analysis of Hacker News posts between Jan 2020 and June 2023

Metaflow sits on top of Maestro, and neither replaces the other

> ...Users can use Metaflow library to create workflows in Maestro to execute DAGs consisting of arbitrary Python code. from https://netflixtechblog.com/orchestrating-data-ml-workflows-...

The orchestration section in this article (https://netflixtechblog.com/supporting-diverse-ml-systems-at...) goes into detail on how Metaflow interplays with Maestro (and Airflow, Argo Workflows & Step Functions)

Metaflow passes state using an object store (s3, azure blob store, etc.) even within Airflow - short circuiting Airflow's xcom machinery. But agreed, Airflow presents many more scalability challenges - this integration addresses a few in-place as well as preserves the ability to swap out Airflow with a more scalable workflow orchestrator if needed.

Outerbounds | Systems dev | SF / Remote | Full-time Outerbounds was founded recently to commercialize Metaflow, an open-source Python framework that makes infrastructure easily accessible for machine learning/data science projects. Metaflow was originally started by us at Netflix and it is now used by hundreds of companies across industries.

We care a lot about thoughtful design, overall product experience, and quality of code. We are looking for backend/systems engineers who are experienced in at least one of the following areas: Delightful API design, deep Python experience, distributed systems, or low-level systems programming.

- Metaflow: https://docs.metaflow.org

- More about us here: https://outerbounds.com/workwithus

- Join our Slack to see the project in action: http://slack.outerbounds.co

- Email: workwithus@outerbounds.co

Metaflow provides a similar concept to interface with Step Functions and Argo Workflows in Python - https://docs.metaflow.org/going-to-production-with-metaflow/...

A single machine can take you remarkably far these days, given the availability of high RAM/Disk/CPU machines in the cloud.

I think another very important piece of the puzzle is *when* to make the transition from a monolith service to a microservice stack. I have seen many startups getting distracted by committing to a microservices transition rather too early in their lifecycle, and on the flip side - big organisations postponing and accruing a huge transition tax.

Metaflow was built to assist in both developing ML models and deploying/managing them in production. AFAIK, TFX is focused on the deployment story of ML pipelines.

https://docs.metaflow.org/introduction/what-is-metaflow#shou...

Give it a few minutes :)

Hi! Metaflow ships with a CloudFormation template for AWS that automates the set-up of a blob store (S3), compute environment (Batch), metadata tracking service (RDS), orchestrator (Step-Functions) notebooks (Sagemaker) and all the necessary IAM permissions to ensure data integrity. Using Metaflow, you can then write your workflows in Python/R and Metaflow will take care of managing your ML dev/prod lifecycle.

https://github.com/Netflix/metaflow-tools/tree/master/aws/cl...

Take a look at metaflow.org/sandbox if you want to test drive Metaflow.

Fixed

I am not familiar with Meltano, sorry.

You’re welcome! :)

Good point. We will address it.

All of our integrations are driven by plugins. We are exploring our roadmap with regards to integrations with other clouds.

Yes, you can use any python library inside a metaflow step.

Metaflow does come bundled with a scheduler that can place jobs on a variety of compute platforms (current release supports local on-instance and AWS batch). In terms of dependencies, we went with conda because of its traction in the data science community as well as excellent support for system packages. Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. In terms of language support, we have bindings for R internally, that we plan to open source as well.

I wouldn’t qualify metaflow as anti-UI. For model monitoring, we haven’t found a good enough UI that can handle the diversity of models and use cases we see internally, and believe that notebooks are an excellent visualisation medium that gives the power to the end user (data scientists) to craft dashboards as they see fit. For tracking the execution of production runs, we have historically relied on the UI of the scheduler itself (meson). We are exploring what a metaflow-specific UI might look like.

As for comparisons with Airflow, it is an excellent production grade scheduler. Metaflow intends to solve a different problem of providing an excellent development and deployment experience for ML pipelines.

Addendum - You can mix and match what steps of the DAG run on the cloud.

Thanks. Feedback noted.

I am not familiar with them.

We erred on the side of simplicity to keep things manageable for our users.

Hi omarhaneef, We don't intend to compete with Tensorflow, PyTorch, SKLearn. What we offer is a way to iterate and productionize your models written using any of the aforementioned libraries (and more). https://docs.metaflow.org/introduction/what-is-metaflow contains further elaboration of our philosophy. Auto-scaling infrastructure is one piece of the puzzle, and Metaflow goes beyond that offering a comprehensive solution for model management.