Hacker News Reader

Comment by Palmik | Hacker News Reader

Why this is interesting: To my knowledge, this is the first time you are able to run the largest Llama, at competitive speed, on a consumer GPU (or something like A40).

On many tasks, fine-tuned Llama can outperform GPT-3.5-turbo or even GPT-4, but with a naive approach to serving (like HuggingFace + FastAPI), you will have hard time beating the cost of GPT-3.5-turbo even with the smaller Llama models, especially if your utilization is low.

View on HN

running 70b on 24gb has been possible for months (so basically forever) it just got like 2x speedup now. the issue that it's a lobotomized ~2bit quant remains.

Quantisation tradeoff is an interesting question. My experience so far is that larger models with lower bit quantisation seem to perform better than smaller models with higher bit quantisation. So, e.g., 30B-4bit appears to perform better than 13B-8bit. But, then, an 8-bit quantisation seems not really distinguishable from the non-quantised model, so what benefit might you gain from not quantising the model.

Of course, speed, accuracy, power consumption, etc are all considerations, sometimes at odds with each other.

It's diminishing returns in terms of perplexity below 4 bits, and this is true even with newer quantisation methods like AWQ and OmniQuant.

I think OmniQuant is notable because it shifts the bend of the curve to 3-bit. While < 3-bit still ramps up, it's notable in that it's usable and doesn't go asymptotic: https://github.com/OpenGVLab/OmniQuant/blob/main/imgs/weight...

What EXL2 seems to bring to the table is that you can target an arbitrary quantize bit-weight (eg, if you're a bit short on VRAM, you don't need to go from 4->3 or 3->2, but can specify say 3.75bpw). You have some control w/ other schemes by setting group size, or with k-quants, but EXL2 is definitely allows you to be finer grained. I haven't gotten a chance to sit down with EXL2 yet, but if no one else does it, it's on my todo-list to be able to do 1:1 perplexity and standard benchmark evals on all the various new quantization methods, just as a matter of curiosity.

Can you do tuning/training on quantized models? Or are they inference only?

This paper[1] attempts to do LoRA fine tuning on quantised LLMs. The LoRa weights themselves are full precision, but they're much smaller. Although, I haven't had a chance to try it yet.

[1]: https://arxiv.org/abs/2306.08162

Yup, but the thing is that can you get model working how you want and can you run it production. My experience is that gpt4 yes, gpt3.5 yes but needs some work Llama2-70b 8-bit or more yes but needs some work after that it really depends usecase.

Something like that 4^30 >>> 8^13?

That's absolutely false, unless you mean running llamacpp with half the layers on the CPU and half the layers on the GPU. And that was like 2t/s.

> running llamacpp with half the layers on the CPU and half the layers on the GPU

How do you do that? Could you please provide a link if you have it handy.

When running you use "-ngl" to specify how many layers are uploaded to the GPU

Just set the --gpu-layers|-ngl flag. Run main with --help to see all the options.

Saying fine-tuned LLaMA can outperform GPT-3.5-turbo on some tasks would be reasonably generous.

Saying it can outperform GPT-3.5-turbo on "many tasks" would be venturing into unreasonable territory.

Saying it can outperform GPT-3.5-turbo or even GPT-4 on many tasks is just setting people up for disappointment: There's a reason ARC (common sense reasoning) never gets mentioned when touting OS LLMs vs commercial, and the gap in ARC performance is amplified terribly when you try to apply chain-of-thought

I am trying to find support for your last argument.

https://paperswithcode.com/sota/common-sense-reasoning-on-ar...

  GPT-3       53.2
  GPT-3.5     85.2
  LLaMa-65B   56.0

Any idea of the performance of an instruction-fine-tuned version of LLaMa models? I can't seem to find non-aggregated performance figures on ARC.

I'm not sure if these ARC scores here are fully comparable to those in scaling, but if they are: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

TigerResearch/tigerbot-70b-chat: ARC (76.79)

Still below 3.5 and like most top entries no clear training objectives, which usually means they were fine-tuned on the benchmark itself.

ARC is easily the most important benchmark right now for widespread adoption of LLMs by laypeople: it's literal grade school multiple choice, created to the bar of an 8th grader.

When you sit people down in front of an LLM and have them interact with it: ARC has by far the closest correlation of how "generally smart" the model will feel.

Wow, the top 14 are all proprietary models (except 2 models which are not commonly used in OS community anyway).

And surprisingly, Llama 33B performs _better_ than Llama 65B!

That table is comparing few shot prompting GPT 4 to zero shot llama.

I am totally with you when it comes to reasoning, or when it comes to generally / universally capable models. But many problems don't require reasoning, and are very concrete / specific.

I use my models primarily for code generation and evaluation. Does the latter part fall into “reasoning” territory?

Advanced code generation, the kind you would need for something like AutoGPT or other agent-like system, is definitely reasoning and GPT-4 or Claude 2 will dominate. But most of the code generation from GitHub Copilot or other systems (like Replit) is much simpler pattern matching (and indeed, Replit afaik uses very small models).

What do you mean by evaluation?

My personal use case is most often supplying files and generating unit tests, as well as supplying a section of code with a prompt about that section to evaluate the approach contextually with some other unseen code. Which perhaps is another form of reasoning.

I doubt a 2.55bit quantised 70bn model will beat gpt3.5 on a meaningful percentage of tasks

Could you explain what you mean by naive approach to serving?

Sure, there are many ways one can be accidentally inefficient, some examples:

Batching: Your throughput (and thus cost per token) will be much worse if you don't do batching (meaning running inference for several inputs at once). To do batching, you either need to have a high sustained QPS or adjust your workload to be bursty (but this has utilization issues).

Inference engine: The regular HuggingFace, even with built in optimizations, is not competitive against inference engines like vllm, exllama, etc.

Utilization: Depending on your scale, it might be hard for you to have a nice flat utilization, or to even utilize one GPU at 100% capacity. This means you need to solve scaling up & down your machines and the issues associated with it (can you live with cold starts? etc.)

Hardware: Forget server-less GPUs like Replicate etc, their markups compared to on-demand pricing is usually >10x. On-demand A100 or H100, provided you can even get a quota for them, are also expensive. Spot instances are better. Older GPUs (A10, T4, etc.) are better. Your own GPU cluster is likely the best if you have the scale (and likely, you can resell your cluster within the next 18months without much if any depreciation).

For these reasons, I have been toying with the idea of providing a dead simple service for fine-tuning & serving open-source LLMs where the users actually own (and can download) the weights. If anyone is interested in this and would like to chat, let me know.

Interested in this. Are you thinking of it being a SaaS type thing? I have some rudimentary experience, i.e. training NanoGPT on both Modal.com and Vast.ai (which is vastly cheaper, but privacy/cost trade off there).

It could be a layer on top of $gpu_cloud (so end user signs up for the cloud) and the service makes use of those resources.

Do you inference NanoGPT with say a Flask app? I made a WebUI for tiny Llama, but it doesn't work for NanoChatGPt

Ha ha not sure yet, I have only done it from the command line. Still got a lot to learn. Building something like this will be good for that!

Thanks for the writeup

Our special case is a single model, high throughput and latency unsensitive.

Can we use multiple lower-memory GPUs, split the the model "horizontally", and pipeline each batch/inference across the GPUs?

Is this a common scenarios handled by the engines you mentioned ?

We’d be interested in helping! aronchick (at) expanso (dot) io

But you can fine tune gpt 3.5 turbo, so your comparison is not clear.

Fine tuned gpt-3.5 is 8x the cost of regular gpt-3.5 for inference, other thread where this is discussed: https://news.ycombinator.com/item?id=37494335

This may be acceptable for some use cases, bot not for others where even regular gpt-3.5 may not be economical.

Reply on news.ycombinator.com