Hacker News Reader

ospillingeron Ask HN: What is your ML stack like?

That makes sense. Programmatically updating cortex.yaml is a common use case especially when you're thinking about continuous deployment. We also have a Python client which can replace the cortex.yaml file (https://www.cortex.dev/deployments/python-client).

ospillingeron Ask HN: What is your ML stack like?

Hey Aaron, I work on Cortex which is a tool for continuously deploying models as HTTP endpoints on AWS. Under the hood we use Kubernetes instead of Lambda to avoid cold starts, enable more flexibility with customizing compute and memory usage (e.g. running inference on GPUs), and support spot instances. Could you clarify your comment regarding editing of config files? Is it still a problem if the configuration is declarative and tracked in git? I'd love to hear your feedback! (GitHub: https://github.com/cortexlabs/cortex | website: https://cortex.dev/)

ospillingeron Cortex: Deploy machine learning models in production

From the MLflow Models docs: "An MLflow Model is a standard format for packaging machine learning models that can be used in a variety of downstream tools—for example, real-time serving through a REST API or batch inference on Apache Spark. The format defines a convention that lets you save a model in different “flavors” that can be understood by different downstream tools."

Cortex is what they are referring to as a downstream tool for real-time serving through a REST API. In other words, MLflow helps with model management and packaging, whereas Cortex is a platform for running real-time inference at scale. We are working on supporting more model packaging formats and I think it's a good idea to support the MLflow format as well.

ospillingeron Cortex: Deploy machine learning models in production

Each model is loaded into a Docker container, along with any Python packages and request handling code. The cluster runs on EKS on your AWS account. Cortex takes the declarative configuration from 'cortex.yaml' and creates it every time you run 'cortex deploy' so the containers don’t change unless you run 'cortex deploy' again with updated configuration. This post goes into more detail about some of our design decisions: https://towardsdatascience.com/inference-at-scale-49bc222b3a...

ospillingeron Cortex: Deploy machine learning models in production

Yes, Cortex uses ONNX Runtime (https://github.com/microsoft/onnxruntime) under the hood so any model that can be exported to ONNX can be deployed.

ospillingeron Cortex – An ML model deployment platform that runs in your AWS account

Good idea, I definitely think it's doable.

ospillingeron Cortex – An ML model deployment platform that runs in your AWS account

The Terraform provider idea is interesting, I'll think about it more carefully. Almost all of our deployment configuration under the hood is done with Kubernetes (which is focused on the declaration of deployment state). We modeled our configuration after Kubernetes for that reason, and we want to go beyond low-level infrastructure configuration by allowing users to configure prediction tracking, model retraining thresholds, and other more ML specific features using the same declarative paradigm and in the same configuration files.

ospillingeron Cortex – An ML model deployment platform that runs in your AWS account

My understanding is that Seldon and Kubeflow are more geared towards infrastructure engineers. Our goal is to hide the infrastructure tooling so that Kuberentes, Docker, or AWS expertise isn’t required. Cortex installs with one command, models are deployed with minimal declarative configuration, autoscaling works by default, and you don’t need to build Docker images / manage a registry.

ospillingeron Cortex – An ML model deployment platform that runs in your AWS account

Thanks for the feedback! We aren't trying to invent another infrastructure provisioning language, and I agree that Terraform would be the right choice if that was the case. Our YAML is more similar to the configuration of deployment tools like Netlify or CircleCI. We use CloudFormation and Kubernetes under the hood but our goal is to provide a much higher abstraction for data scientists / ML engineers.

ospillingeron Cortex – An ML model deployment platform that runs in your AWS account

We solve a similar problem to SageMaker but we are focused on developer experience and flexibility.

- Deployments are defined with declarative configuration and no custom Docker images are required (although you can use your own if you want)

- You have full access to the instances, autoscaling groups, security groups, etc

- Less tied to AWS (GCP support is in the works)

- We are working on higher level features like prediction monitoring, alerting, and model retraining

- It's open source and free vs SageMaker's ~40% markup