Hacker News Reader

Comment by alchemist1e9 | Hacker News Reader

I’ll ask a dumb question. On another of the numerous LLM related posts I was asking if any of the self host-able open model can do code summaries at close to the quality of GPT 3.5 turbo. I was basically told nowhere close yet.

Can this potentially do that?

Ideally I’d like to have it generate descriptions of large amounts of code but would rather not burn tokens and lose privacy via OpenAI api. But I’d gladly keep a high end GPU burning on such task , even if that was actually slightly more expensive.

Edit: To clarify I do this partially now on batches of code via openAI api currently. It’s around 1-3 cents for a typical 400-800 line source code file. And I don’t mean feeding the full code base in as a single input.

View on HN

Here's my experience, having used llama+lora 7b, 13b, and 30b, on both cpu and gpu:

On gpu, processing the input prompt, even for huge prompts, is almost instant. Meaning, even if your prompt is huge, it will start generating new tokens after your prompt very quickly. On a rented A6000 gpu, using llama+lora 30b, you can use huge prompts and it will start giving a new output right away.

On cpu (i.e. the project llama.cpu), it takes a very, very long time to process the input prompt, before it begins to generate new tokens. Meaning, if you provide a huge copy/paste of code, it will take a long time to ingest all that input, before it begins outputting new tokens.

Once it finally starts outputting new tokens, the rate is surprisingly fast, not much slower than gpu.

I wish I knew the reason for this, but I'm not an expert :) I've just seen this in practice.

That sounds promising but how about the quality of the output.

I’ve been using OpenAI API with chatblade and been giving it code in various languages and it’s quite surprising how well it describes the purpose and code implementation in english. The english description would be useful and relevant for developers trying to quickly familiarize themselves with a code base.

For a typical 400-600 line file it looks like it would cost around 1-3 cents per file. However loss of privacy isn’t so great.

How do you find output quality of llama + lora for such a task?

Code is research code I’d run it on.

How long is very very long? Am I going to get coffee while it works, going to lunch, doing it right before I go to bed, or hoping it finishes in time to come up with the most perfect epitaph on my tombstone? ;)

That is likely a bug. There is nothing in the maths that should make prompt tokens slower on CPU.

It is a bug, it's been clearly discussed under the issues of LLaMA.cpp. The wait time is far from atrocious, with a 5600x and the 13B I would wait 15 seconds before input starts for a relatively complex prompt.

Any tips on setting up llama+lora on 30b? There are so many resources that I can't figure out which models to use, and which projects to use to set everything up.

text-generation-webui (https://github.com/oobabooga/text-generation-webui)

I tried it, but I can't find the right model for llama+lora 30B . Google, nor Bing ;) is not helpful.

I think this is the one you want: https://huggingface.co/elinas/alpaca-30b-lora-int4

As I understand these models, that makes no sense whatsoever. Is it the tokenisation that takes this crazy amount of time? Because each additional token should take exactly the same amount of time as the first.

We're a long long long way off that. So check back in two months.

Jokes aside, the limiting factor will be either a technique to pack all of the code into smaller tokens like semantic search (someone else will be able to comment on this as that's the limit of my understanding) or GPU memory for input tokens.

Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.

Are you sure?

This is for inference, not training, and I’d do it on blocks of code with less than 4096 tokens. That’s what I have to do with gpt 3.5 turbo via api.

I can do what I want with the API and it is around 1-2 cents per batch of code I run through it, typically 400 lines of average code is around 4k tokens.

If you can get it within 4k of tokens then your vram needs will be much less. As the other two have pointed out, there's workarounds to this.

If you're trying to get codebase level introspection you might need to wait a bit for some of these techs to mature.

Exciting space and yea, learning heaps myself day by day.

The approach I was going to take is generating english code descriptions/summaries in related chunks of under 4k tokens then those can be combined and also summarized.

In my opinion the code descriptions gpt 3.5 turbo has been spitting out for me are good quality and concise. I’d argue they are probably better than what many of the developers themselves would write, especially when english isn’t native for the developer.

It's $0.590/hr to rent an A6000 48GB gpu on jarvislabs.ai, which you can do right now, and run the 30b (or 65b with some hard work) model and get incredible results. No confidentiality agreement required :)

Do you know how their "spot pricing" works, exactly? "lets you use spare GPUs" doesn't tell us much. Is it granular to how long the duration the model is actively processing your prompt? ..or do you get dinged for an hour immediately and again on the hour?

>Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.

I haven't started to play with LLMs locally in anger yet, but I was under the impression that you could use a 4090 in combination with FlexGen to achieve this rather than having to buy special hardware?

With 8 bit training you can do ~13B pram LLM on 3090/4090. https://huggingface.co/blog/trl-peft

But it is pretty cheap to rent something at vast.ai or whatever to get 40GB for a final run.

Awesome. Between crypto hype in 2017 and AI hype in 2023 I've acquired a collection of 2x 1080ti, an RTX 3060 and an RTX 4090. All together it's a total of 58GB of VRAM. Is there a way I can pool it all across a distributed cluster of 2 machines for doing anything? I'm assuming it would bottleneck on both network speeds and the slowest GPUs if it's possible at all...

Could you package all that compute into a "virtual" graphics card?

"Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k."

Does anyone have a projection based on historical GPU cost reductions as to how long we have to wait until a DGX unit costs as little as a 4090 does today?

The issue is largely around available RAM capacity, and sadly RAM price have barely decreased in the last 5-10 years

There is ChatGLM[0], a 6 billion parameter Chinese/English bilingual model that is gaining a reputation as the leading locally runnable LLM for code generation. Maybe look into that. Demo is here[1].

[0]https://github.com/THUDM/ChatGLM-6B/blob/main/README_en.md [1]https://huggingface.co/spaces/multimodalart/ChatGLM-6B

I tried this the other day for generating basic Chinese conversations - the quality is surprisingly good. While it's still behind the latest GPT, but the gap isn't as large as I thought it would be.

This does seem promising when playing with the demo. That says running on a T4 I believe. This probably means we are only months away from viable self hosted code generation and analysis that could compete with gpt 3.5 turbo.

It seems like working on locally runnable LLM that has been fine tuned to focus on code should be a high priority.

I'm doing the same tasks but on english text, I have not found a local LLM anywhere near the summarisation ability of GPT 3.5 and 4.

That said, there are some summarisation tasks I prefer to run locally even given the massive drop in quality (e.g. using Alpaca), both for privacy reasons and to keep myself up-to-date on local LLMs.

P.S. I'm discounting older local summarisation-specific networks, I have found local LLMs to be a jump in quality over them

Same question here - alpaca does not do well with long inputs :( Ideally i can throw a 100 pages PDF at it and get a summary and a response document :)

Can’t you just break it into chunks and the summarize the summaries?

Think long lawsuits and discovery. Possible to break it up but really not ideal.

Yeah I think you just need a tool built with Langchain and it can do what you want.

Care to share what prompts you run this with? What is the output you desire, and how do you achieve this?

Prompt: You are an expert python programmer. I will give you some python code below. I want you to describe the code and it’s likely purpose.

The answer will be pretty damn good , especially with gpt 4, but it’s still good with gpt 3.5 turbo. Try it out yourself with whatever code you have.

This english description of code can then be searched against. That’s the primary practical application I’m thinking about doing currently.

Reply on news.ycombinator.com