Hacker News Reader

Comment by SparkyMcUnicorn | Hacker News Reader

I haven't fine-tuned GPT-3.5-turbo yet, but overall llama2 is probably going to be cheaper. Especially if 13B is good enough for you.

You can get roughly $0.003/1K tokens using modal.com for inference on a fine-tuned llama2 13B model, and there's definitely cheaper options available.

Unless you have a ton of data, the cost for training llama2 is usually a few dollars.

View on HN

I'm curious about this. Can you point me to, e.g. some example code for setting up an inference endpoint with a base llama2 model on modal.com?

Here's one if their tutorials using vLLM, and they have a few other guides and example repos as well.

https://modal.com/docs/guide/ex/vllm_inference

https://github.com/modal-labs

Alternatively, Runpod is fairly cheap and easy to get stuff running in a few minutes and can be point/click only using their templates.

https://www.runpod.io/console/gpu-secure-cloud?template=f1pf...

("serverless" example) https://github.com/ashleykleynhans/runpod-worker-oobabooga

Thanks for that. I've used RunPod GPU cloud to setup vLLM as an Open-AI API compatible endpoint before, but haven't tried any of the serverless options yet.

> the cost for training llama2 is usually a few dollars.

nit: thats the cost for finetuning, not pretraining

Yeah, definitely not pretraining. I've been referring to fine-tuning as training, since that's what's happening when you "fine-tune" a llama model.

Reply on news.ycombinator.com