Hacker News Reader

Comment by blodovnik | Hacker News Reader

Aws and Google spot instances are easily comparable price wise to hetzner but it really depends on what you are doing, which you haven't said.

For example a g3 spot instance can be as low as 19 cents per hour $138/month.

Google GPU instances I've run for 14 cents an hour.

I think these aren't the lowest prices either.

View on HN

For one-time GPU processing tasks that only take a couple hours, AWS spot / GCP preemptible is far more cost efficient than Hetzner.

However, if you need to run a GPU 24/7 and/or have massive bandwidth requirements, Hetzner is far more cost effective than AWS/GCP.

It's a balance of needs.

Another option is to over provisioned spot instances, such that it would feel like a 24/7 instance, but cost like spot. However, you would probably need an automatic platform to achieve that.

An idea I've had is to use GCP preemptibles + script to automatically run a ML training job on instance start + Google Cloud Function + Cloud Scheduler to attempt to start the instance every minute if it gets preempted. The latter two are effectively free, so you'd get the cost benefits of preemptibles as long as the ML training job is resilient to random shutdowns.

Yes, this is basically the idea. However, there are different solutions for training and inference. For training, I would recommend that you add automatic checkpoint, and even consider model migration. For inference (which I think is the original concern), over provisioning is the key (simply because the fact that it would take a long time to load the model. Also, you also want to diversify your node types, etc.

AWS spot instances for gpu can have availability issues as well or sometimes be priced higher than on demand (still trying to understand that one).

If you do choose to run spot at aws (gpu or otherwise) be sure to check out the excellent project at autospotting.org and donate if you use it. Makes it super easy to replace on demand nodes in an ASG with spot nodes and always make sure you’re getting a good price.

> AWS spot instances for gpu can have availability issues as well or sometimes be priced higher than on demand (still trying to understand that one).

You always pay the spot market price, not your bid. Your instance gets killed if somebody outbids you. A higher bid increases the probability that your instances don't get killed while at the same time letting you pay spot market price.

By bidding above on-demand price, you are speculating that for the majority of the time, nobody else will bid more than the on-demand price. If you're not the only one doing that, spot price can rise above on-demand price.

The main use case is on demand neural net inference 24/7 availability.

Kind of like hosting a website that must be on 24/7, this neural net must be on 24/7.

It runs on the order of 50x faster on an nvidia GPU compared to a CPU.

But it will sit idle most of the time.

Price per hour per GB of GPU ram is the most important metric when choosing a server.

I really like nocix servers, they are out of Kansas City. They got their "i7-6700K 32GB + 2x 480GB SSD + GTX 1080" for $105/mo https://www.nocix.net/cart/?id=338

This sounds great but they are also out of stock!

Would AWS Elastic Inference work for your use case?

https://aws.amazon.com/machine-learning/elastic-inference/

No, I need NVIDIA CUDA support. Also, the most important metric for me is cost per month per GB of GPU RAM, and AWS elastic inference is pretty bad in that metric.

Reply on news.ycombinator.com