Show HN: Dstack – a command-line utility to provision infra for ML workflows | Hacker News Reader

Show HN: Dstack – a command-line utility to provision infra for ML workflows

github.com

5 pointscheptsov8 comments

Hi. :) I’m Andrey, the creator of dstack. I started this project while I was working at JetBrains where I helped the PyCharm team to improve support for Jupyter notebooks. As I was in close contact with many ML devs (who used PyCharm) I was able to see their struggle with running ML workflows.

Unlike traditional dev workflows, ML workflows are difficult to run on a local machine (due to the lack of memory, more CPUs/GPUs, etc). This is why people often have to use remote machines (e.g. via SSH), or adopt one of the end-to-end MLOps platforms.

Using remote machines is not difficult but it is tedious and requires a lot of manual actions. Using MLOps platforms on the other hand automates the manual work but requires the use of an opinionated interface, which often kills developer productivity.

Imagine, if you could run your ML workflows the very same way as you do it locally, but they would actually run in the cloud. And you wouldn’t need to worry about provisioning infrastructure, setting up the environment, etc.

I’m excited to show you dstack, an open-source tool that does exactly that.

It’s a command-line utility that allows you to run any workflows while it provisions infrastructure, setup the environment, and copies code/data for you. No need to install or configure anything in your environment or cloud. Simply install the CLI and run it.

The launch blog post: https://mlopsfluff.dstack.ai/p/simplifying-the-mlops-stack

We’d love to hear your thoughts and ideas. I’ll be here to answer any questions you might have.

Open articleView on HN

I have a task:

URLs to 10M images.

I need to create a thumbnail for each of them.

I.e. I need to download each image, resize and save in the webp format to the S3.

Right now I am doing this locally and it takes forever, as my images are in high resolution their total size is a few Tb.

Is there a tutorial that will describe how to do it? I was thinking about using AWS Lambda + SQS, but if DStack is an easier solution, I would be happy to check it out.

Thank you for the comment! Exactly, this is one of theses tasks, dstack is designed to help with. You basically write a simple Python script that does the job, tests that it works locally, and then run via dstack – asking for a more heavy machine.

The entire setup will take roughly 5-10 mins. Promise to add a tutorial with exact steps and code snippets soon.

In the future, we also plan to support distributed workflows. Actually, this is already supported by the design, we just need to implement a provider!

Hey, congrats :) It's good to see more tools in this space. Question - how does it handle datasets and data management in general? Let's say we have a large amount of files on S3. Would it my task to copy them first as an initial step?

Thank you! That's a great question. First and foremost, dstack treats artifacts are 1st class citizens.

Here's the basic way of using them:

A workflow may produce output files to a local folder. In your workflow declaration, you can mark what folders to treat as output artifacts. Then, dstack would save output artifacts automatically, and you'll be able to reuse them via the unique name of the run, or a user tag assigned to this run.

All artifacts are stored in the S3 bucket that is configured for dstack. dstack is capable of syncing artifacts at start//end of the workflow or mount artifact folders via FUSE (of course if that is needed).

Each artifact is stored using the following path: <s3 bucket>/artifacts/<run name>/<job name>

A run can have multiple jobs, e.g. if it's a distributed workflow.

In future, we also think of providing a high-level Python API for accessing/storing artifacts.

Please share your thoughts and feedback!

Yes, by default, it will copy the files. Or you could also tell dstack to use FUSE.

As I said in the comment above, we think of providing a high-level Python API. THis API will allow to avoid copying the artifacts as an initial step.

But of course would love to hear what you think! ;-)

This looks really interesting. Does it turn off the env after its done, cost would be a factor here when trying this out.

Hey! Sorry for late reply. Yes, it does destroy the resources once finished. Also, it allows you to use interruptible (aka "spot") instances efficiently.

Awesome thank you.

Reply on news.ycombinator.com