Hacker News Reader

Even GPT3.5 can be much more expensive. In some specific tasks, a finetuned 7B llama can work as well as GPT3.5.

You can rent a 3090 at $0.20/h on vast.ai, or $0.40/h on runpod. Using VLLM at 400t/s that's 1440000 generated tokens. Generating that amount of tokens with GPT3.5 would be $2.88.

View on HN

> "In some specific tasks, a finetuned 7B llama can work as well as GPT3.5."

"some" is doing a lot of heavy lifting here.

Also: don't discount the labor cost of curating a fine tuning dataset, running a FT training run, even if the hardware is cheap.

Unless your use case isn't in English in which case LLama is as useful as a one-legged man in an ass-kicking contest.

LLama models only really shine for things that GPTs would refuse to even consider because of corporate RLHF, and if you need to keep your data local I suppose. For the rest they're second rate at best.

Reply on news.ycombinator.com