Hacker News Reader

Hetzner do cheap servers, but don't have many GPUs. If you're looking at saving money by running on CPU, you shouldn't be looking at one of the most expensive server providers.

View on HN

How is the $ per inference, say 4k tokens, on a Hetzner box vs an A100?

Not all tasks require low latency.

I'd like to know the answer to that too. I shouldn't have implied that CPU inference on Hetzner is cheaper than GPU on AWS, when I don't have any idea on the cost of either.

Hetzner offers incredibly cheap ARM machines in the Falkenstein DC, for 25Eur a month you can snag the top of the line with 16 vCPU and 32GB RAM.

If your usecase fits inside that 32GB (no 70B models, sadly) the price to performance of a GGUF Q4KM is really attractive on this setup.

With two/three instances, you can probably fit a 70B model into RAM, and you don't need super low latency between models to be able to do inference split layerwise between machines.

Are there instructions for this distributed inference somewhere? Can I do this out of the box with llamacpp or similar?

Don't think so. I suspect it would require quite in-depth surgery of llamacpp to add in the ability to send activations over the internet and pipeline stuff to keep all the cores busy.

Reply on news.ycombinator.com