Show HN: SpotML – Managed ML Training on Cheap AWS/GCP Spot Instances

Congrats! Can I suggest charging more?

IIUC, your business plan is $10/month for the company, regardless of number of users?

You probably save a company that $10 in a day or less for one GPU: One A100 is ~$3/hr on demand and ~$.90/hr as Preemptible, saving over $2/hr.

Said another way, your pitch is to recover a lot of the 70% discount that they aren't going to do themselves. If you were a managed training service, you could pitch yourself as "half the price of AWS or GCP" and keep the 20%+ margin with both parties being happy. (The problem is that pass through billing makes that obvious, you need to support lots of bucket security and IAM controls, etc.).

Fwiw, I would also branch out into inference! Preemptible and Spot T4s are commonly used for heavy image models, but many people pay full price. Inference that takes X ms can easily be handled "without errors" in the shutdown time. The risk is handling all the capacity swings.

Spot Instances are 70% cheaper than On-Demand instances but are prone to interruptions. We mitigate the downside of these interruptions through the use of persistence features, including optional fallback to On-Demand instances. So you can optimize workflows according to your budget and time constraints.

History: We were working on a neural rendering startup that needed a lot of GAN training which was getting very expensive. We were blowing roughly $1000, to train a single category class. Training on Spot instances was cheaper, but still a mess. It needed lot of hand holding/devops stuff to make it usable. So we built SpotML to automate a lot of things.

Posting it here to see if the community finds this helpful, so that we can open it up to the larger community.

Nimbo docs: "In order to run this job on Nimbo, all you need is one tiny config file and a Conda environment file (to set the remote environment), and Nimbo does the following for you:"

SpotML docs: "In order to run this job on SpotML, all you need is one tiny config file and a Docker file (to set the remote environment), and SpotML does the following for you:"

Make of that what you will :).

Also there's at least two open source free solutions for elastic training I know of. RaySGD, https://docs.ray.io/en/master/raysgd/raysgd.html and elastic horovod, https://horovod.readthedocs.io/en/stable/elastic_include.htm... For me to consider this I'd need a comparison table with native framework solutions and these solutions along with what does it add for me.

Why not just train on Spot Instances with a retry implemented?

I see that SpotML has a configurable fall back to On-Demand instances, and perhaps their value prop is that it saves the state of your run up to the interruption + resumes it on the On-Demand instance, but why not just set a retry on the Spot Instance if its interrupted?

I'm failing to see what is different about SpotML vs Metaflow's @retry decorator and using AWS Batch: https://docs.metaflow.org/metaflow/failures#retrying-tasks-w...

If you're in the comment still, Vishnu, would love to hear your thoughts

I used to have these same pains. My trick with spot instances has been to set my maximum price to the price of a regular instance of that class or higher and to sync weights to S3 in the background on every save. The former is a parameter when starting the instance in the console or terraform, the latter is basically a do...while loop. I've noticed that often one gets booted from a spot instance causing interruption because the market price increases a few cents above the "70% savings price." Increasing the maximum to on demand is basically free money because you don't get booted often, and your max price is the regular on demand price.

This has seemed to mitigate about all of the spot downsides (like interruptions or losing data) because you don't easily get kicked out of the spot unless there's a run on machines and the prices rarely fluctuate that much (at least for the higher end p3 instances). This has seemed to prevent data loss and protected the downside risk by setting the instance to a knowable max price. There are instances where the spot price goes higher than the on demand price, so you still get booted once in a while, but it's very infrequent as you can imagine most people don't want to spend more for spot than on demand.

Anecdotally I still average out to getting the vast majority of the spot savings with this method with very few interruptions. Looking at the SpotML it seems to be a lot of tooling that achieves these same goals (assuming one would be interrupted when a spot dies and moves to full-freight on-demand with SpotML), which makes SpotML's solution feel very over-engineered to me given that the majority of what SpotML provides can be had with a simple maximum cost parameter change when spinning up a spot instance.

I would be very interested in using anything that doesn't have great overhead and saves money. Our bill seems "big" to me (but I realize it may be small to many others), so even these small savings add up. Would you compare the potential benefits of SpotML to the method I described above?

I hadn't heard of nimbo, maybe I can read how they're doing it since it's open sources. Does anyone have any idea how they're saving state so fast (NVME SSD disk?)

Source: https://cloudoptimizer.io

Seems like the services I've tried, have focused heavily on supporting notebooks (colab, paperspace).

Any ideas?

Could you give an approximate cost of fine-tuning a model like Bert or even GAN given this system?

Just want to get a sense of cost of using the system.

All you need to do is schedule your jobs (just call the Kubernetes API and schedule your container to run). In the rare case the node gets preempted, your job will be restarted and restored by Kubernetes. Let your node pool scale to near zero when not in use and get billed by the _second_ of compute used.

unable to figure out how to construct the job itself and how to submit the dockerfile to be executed.

also - do u support distributed training ?