Ask HN: Cheaper way to do model inference?

2 pointsmasterofall26123 comments

Does anyone know of any solutions for saving GPU compute during server downtime? Is there a managed solution to turn off a pod and turn it back on when I need it? I'm currently doing model inference and most of the time I'm just paying for compute without serving any user requests.

View on HN

Huggingface Inference Endpoints can autoscale to 0 and cost nothing when not being used.

You are running inference on something like an EC2 instance?

Yea exactly that's what I'm doing right now. I've looked into solutions like RunPod serverless, anyone have any experience with it?

Reply on news.ycombinator.com