Show HN: Self-host open-source LLMs on AWS with scale-to-zero | Hacker News Reader

Show HN: Self-host open-source LLMs on AWS with scale-to-zero

github.com

7 pointspaguasmar5 comments

Basically the CLI

1. compiles your model code and dependencies without requiring Dockerfiles or K8s

2. provisions spot or on-demand instances directly in your own AWS/GCP account (via SkyPilot)

3. it then spins up the runtime and gives you a production-ready URL endpoint

I wrote this originally to replace Modal and Baseten, which I wasn't too pleased with. I needed to deploy open-source models inside my own VPC without vendor lock-in or proprietary Python decorators.

Thought the community here might like it. AWS is an amazing cloud - with one-command deployment and scale-to-zero I was able to consolidate all my LLM serving into my own infrastructure for a fraction of the cost.

Would love feedback. There are a few (many?) bugs and a lot of things to iron out, but I'm working with some close friends to make it awesome.

You got me interested, a rough table of model size to instance type to spot $/hr would help a lot. The 0.5B example is CPU only, so it doesn't say much about what a 7B or 70B actually costs.

You're right. A CPU only model is definitely is not a real prod workload where you'd need GPU machines.

Here is a rough table of model size to instance type to spot $/hr:

For up to 8B, you can use a c6i.xlearge which costs as spot about $0.4/hr.

For up to 70B, you can use a g5.12xlearge that costs you about $2/hr.

Besides the GPU machine be aware that you need a controller instance to route the requests and scale. That has a fixed cost of up to $0.08/hr. When not in use at all, just type veloxml down --all and it tears down the controller too for true $0/hr.

We're currently running larger LLM models benchmarks to add to the README this week.

What happens when AWS interrupts a Spot instance? I don't see an ASG in the architecture — does the controller detect the terminated worker and provision a replacement automatically?

Great question. The controller provision a replacement automatically. The controller runs continuous health-check polls against the worker’s, if it doesn't respond, provisions a new one.

Why not use a AASG?

I chose to use a controller to make the solution cloud agnostic.

Reply on news.ycombinator.com