Hacker News Reader

Comment by ogou | Hacker News Reader

I rolled my own version of this for a side project. I ended up on a Digital Ocean droplet with RAM upgrades.

https://lucidbeaming.com/blog/fine-tuning-a-gpt-2-language-m...

The extra monthly $40 for a novelty project has me searching for cheaper alternatives. Google preemptibles scared me off because they can scale dynamically and generate runaway costs if my project suddenly got popular on Reddit or something.

View on HN

Preemptible instances should not have any associated auto-scaling. It is common for people to run pre-emptible instances as part of a "Managed Instance Group", with which you can associate some kind of autoscaling policy.

You can run preemptible instances without any kind of management or with only very simple management and you should be okay. I recommend a simple managed group with a policy of, "I want maximum one of these instances running at all times and, if it gets preempted, then please start a new one."

If you're using Google Cloud, don't deny yourself the cost savings of preemptible instances. When I was using the same infrastructure, preemptions were very rare!

Nice! Thats a cool side project. Not sure how Google preemptible can scale though. They are not auto-scaling - so its a fixed cost

I swear I saw something in the terms about charges from sustained uptime, which is related. A popular resource would trigger that. It starts and stops on demand so that automation is what I am leery of. I will look into it again though, maybe I misinterpreted the fine print.

Ah no thats more for AWS spot instances I think. In Google, preemtible rates are constant, which makes it way more deterministic

What's the average inference time like for your model on DO?

65 seconds

This is now 240 seconds on a Google Cloud N2D instance. I successfully recreated my inference worker on a Google preemptible instance. My monthly cost went from $40 on Digital Ocean to ~$16. It's much slower though.

Nice big cost reduction! Is this using BudgetML from the post? Have you tried optimizing the model (i.e quantization and converting to something like ONNX)? I know this can bring big speed gains on T5 on CPU (5x faster), another generative model. More info here: https://discuss.huggingface.co/t/speeding-up-t5-inference/18...

Reply on news.ycombinator.com