Hacker News Reader

Comment by thumbsup-_- | Hacker News Reader

Can anyone share their thoughts on how does this compare to fine-tuning Llama2 13B or 70B which usually requires renting GPUs from one of the cloud providers?

In case of openAI, the infra doesn't needs to be rented but the cost is covered in usage pricing but want to hear from someone who has experience fine-tuning Llama2 models that how it compares.

View on HN

I'm not an expert on this subject, but I have done a bit of gpt3 finetuning through their api:

I think it's clear that "fine tuning" with GPT is different from fine tuning something like Llama2, in that it probably isn't adjusting all the weights of the network, only a tiny subfragment of the network- Exactly how OpenAI accomplishes this is properietary.

The tradeoff is that OpenAI fine tuning is less expensive, but it is also less powerful than "real" fine tuning.

> it probably isn't adjusting all the weights of the network, only a tiny subfragment of the network

source please? this actually isnt all that clear to me

It was what I read on forums when I learned about the process. It's possible that I am mistaken.

It’s almost certainly just a LoRA

Not an adapter?

I've been taught in many cases you can indeed fine-tune the last (i.e. closest from the output) layer(s) of a network. Of course, it does not give as good results as fine-tuning the whole model, but it is obviously way less expensive in compute.

i.e. you actually don't want your model to re-learn _everything_.

I haven't fine-tuned GPT-3.5-turbo yet, but overall llama2 is probably going to be cheaper. Especially if 13B is good enough for you.

You can get roughly $0.003/1K tokens using modal.com for inference on a fine-tuned llama2 13B model, and there's definitely cheaper options available.

Unless you have a ton of data, the cost for training llama2 is usually a few dollars.

> the cost for training llama2 is usually a few dollars.

nit: thats the cost for finetuning, not pretraining

Yeah, definitely not pretraining. I've been referring to fine-tuning as training, since that's what's happening when you "fine-tune" a llama model.

I'm curious about this. Can you point me to, e.g. some example code for setting up an inference endpoint with a base llama2 model on modal.com?

Here's one if their tutorials using vLLM, and they have a few other guides and example repos as well.

https://modal.com/docs/guide/ex/vllm_inference

https://github.com/modal-labs

Alternatively, Runpod is fairly cheap and easy to get stuff running in a few minutes and can be point/click only using their templates.

https://www.runpod.io/console/gpu-secure-cloud?template=f1pf...

("serverless" example) https://github.com/ashleykleynhans/runpod-worker-oobabooga

Thanks for that. I've used RunPod GPU cloud to setup vLLM as an Open-AI API compatible endpoint before, but haven't tried any of the serverless options yet.

Reply on news.ycombinator.com