Run 70B LLM Inference on a Single 4GB GPU with This New Technique | Hacker News Reader

Run 70B LLM Inference on a Single 4GB GPU with This New Technique

ai.gopubby.com

111 pointsgardenfelder57 comments

“This new technique” is they load each layer into memory, process the result of that layer, and then clear it and load the next layer. For the cases where this even works, as one commenter said:

“I try it on a GTX 1060 6Gb on windows, but I don’t think it’s supposed to be that slow? It took me 13 hours to generate one sentence, RAM was around 9.5Gb and only 6% of GPU in use. It works but I don’t know what I’m doing wrong”

> “This new technique”

Which also has a name that they even use, model sharding. Which is well known to be very slow because you increase the number of IO operations to the GPU. I don't even understand the post because they mention sharding is slow and then say they do sharding but as if they don't. Even the comment in their code calls it sharding. Unless I'm missing something, it's pretty fucking deceptive to call this a new technique. Can we stop with the AI hype? The field is exciting enough that we don't need to be selling snakeoil and selling everything as if it is 10x better than it is.

For anyone interested in model sharding you should look into pytorch's gradient checkpointing documentation and/or fairseq.

To echo and in no way contradict the parent’s point, there have been versions of this going on forever.

Without getting into a bunch of stuff about FDSP or Accelerate or even the NCCL/All-Reduce type ops (which I obviously hope the interested check out) I’ll signal boost my favorite tech talk ever by maybe the coolest CS prof living:

https://youtu.be/l5JqUvTdZts?si=9HlNWHTlRbiatHD1

I’m biased because I’m also Ben, but I’m not “Ben Rekt” who is a musician and charismatic and way too cool to care about SV.

Btw, that link isn't working for me, this is though https://www.youtube.com/watch?v=l5JqUvTdZts

Hogwild is a really cool routine. There's a lot of work going into FDSP, parallelism, and all kinds of things at the low level for optimizing communication, maximizing parallelization, writing cuda kernels, and so on. I think there's a lot of unsung heros of ML working in these areas that people never even see (including researchers). So many people quote Sutton's Bitter Lesson but these are the people that made that landscape even possible. Without these people working at the low level we wouldn't be able to process large models or large datasets.

The fame issue is always a bit weird to me. Like I think more people know of Vaswani (AIAYN) or Dosovitskiy (ViT paper) than those that know Kingma, who wrote the Adam optimizer. Both these works use Adam as well as Rombach's LDM (Stable Diffusion). I'm only mentioning Kingma because I'm more aware of that work but am certain there's tons of work that could be better highlighted (actually illustrating this point).

I was playing with this with the llama model was released - and provided I was loading from memory and not disk it was actually only negligbly slower than the model was without sharding (for e.g. the 8gb models that I could load entirely on my GPU). Each layer took a lot more time to evaluate than it took to copy the weights onto the GPU (RTX 2070 super).

Yes, if you can fit it in RAM then the loss isn't too bad if you have the memory pinned and are taking advantage of some of Nvidia's optimizations (which should be defaulted to on). Inference usually isn't too bad but training is heavier. Things have also massively improved over the last few years. The efficiency also really increases with reduced floating point types and quantized models.

Yeah, I'm sure the performance of this must be pretty bad. As I understand it, non-batched LLM inference is generally limited by the bandwidth required to read the entire weights from GPU memory for each token produced. This trick instead moves the bandwidth bottleneck to loading those weights into the GPU from wherever has enough space to store them, probably some kind of SSD, and that has much lower bandwidth than GPU RAM.

For CPU inference, performance is generally memory bandwidth limited. A modern cpu can do the matmuls (all of which are matrix-vector) faster than the data can be loaded in from memory.

If the model is all in RAM and being moved one layer at a time to a GPU, would that be faster than getting it to the CPU? What about if the model is on an SSD. The point being, compute is not the bottleneck so I wonder if this provides speedup compared to an optimized model run on a CPU?

Tried this awhile ago on the llama models, with my hardware

1. The GPU was still faster, though the difference was smaller than I really expected.

2. Almost all the time was spent on compute, not loading weights to/from the GPU. For models I could fit on my GPU the difference in performance in loading one layer at a time vs loading them all was completely negligible, and the difference in memory usage (constraining what else I could do with the computer simultaneously) significant.

3. Loading from SSD was a huge bottleneck when I tried that

Hardware: RTX 2070 super, 5800x

Just want to add, this approach is the exact opposite of conventional wisdom around cache hierarchy. Here we are loading data in from the slowest source, using it once, and throwing it away. I'm not sure what can be changed with a sequential model like and llm, just that this is almost the textbook slowest way to run the model.

It would potentially be slower because pcie bandwidth is likely to be lower than memory bandwidth.

The way to get a speed up is to split the load across both the CPU and GPU since LLMs can be split into layers. This also increases the size the model can be.

It sounds like they do it naively, which is fair for a POC

Even for non-batched, you can still optimize this a bunch for fairly big models without much more work. Basically, overlap io & compute: while one layer is computing, already be loading in the next layer(s) in parallel. PCI speeds are in the 1GB/s-32GB/s each way for consumer cards nowadays.

Without a lot of work, they might be able to fully hide the IO latency.. very model + hw dependent.

I appreciate your and the other commentators scepticism, and agree for the premise of generating text. However to play devils advocate - Can't inference of single layers be quite useful sometimes for embeddings? and the larger models could embed capabilities only achievable at large parameter size? (i.e. you don't need to finetune the last layer like the BERT days or necessarily use a certain layer for relevant embeddings, you could use the first layer outputs and finetune a custom matrix projection to make some useful embeddings that might include properties of the input that the larger models show capabilities for that smaller models don't?)

(ofc this assumes that those capabilities considered "emergent" at larger sizes do not require later layers which they probably do). There is a weird situation where we have the sentence transformer, the bge, e5 and etc kind of embeddings, and then a big jump up to generative model embeddings the model providers provide, but not much widespread adoption of e.g GPT-J or neo-20B embeddings (even though at time of release, they had notable usecases over sentence transformers)

There must be some real use for it for sure. Maybe as a result of this effort someone else has their attention called to it and picks up the idea loosely and runs with it in an unexpected way.

One of the benefits of lack of contextual or historical understanding is the lack of barriers to action. One of the downsides is you’re likely to do a lot of things of questionable value.

You can save some space, but it still takes more than one layer to do an embedding.

> 13 hours to generate one sentence

This is hysterical, they're literally swapping from VRAM to RAM and certainly to disk as well since a 70B GGUF is at least 30-40 GB in total.

I suppose it's slightly better than not being able to to it at all, but jesus christ this is just hilariously bad.

Note how they carefully avoid naming an actual tokens/second number for any gpu.

Are there any resources that do provide this? I’ve tried on numerous occasions to search for stats on model XYZ puts out x tokens/sec on this hardware setup. It leads me to believe the only way to experiment with running open models is with cloud computing and expensive GPUs and consumer hardware is out. Even for 7B parameters.

7b can run fine on consumer hardware if you quantize to 4bit. I get around 50-100 token/s on 7b q4 models on older 8gb gpus I have lying around, which is plenty for chat. I don't know of any good resource for "consumer hardware leaderboard" though. In general:

1.3b is about the max you can get for 4gb vram

7b will work on 8gb vram if you quantize

13b will work on 16gb vram if you quantize

33b can work on 24gb if you quanitze

If you lack vram but have ram and infinite patience you can do offloading shenanigans with cpu but it tanks performance to unusable levels in my experience.

Considering quantizing takes a while, care to list some settings guide or a link for people to setup?

I use Phind_34B for a 24GB gpu. When in use I typically set gpu memory limit to 22.5GB.

Quantization: 1hr 3min. 4bit, group_size=32, desc=true, damp=0.01, trust_remote = false, dtype torch.float16, use_triton = false.

I'd defer to the llama.cpp or exllamav2 repos for quantizing help, I just use their guides myself. For practical use, I'd say searching for the quantized model on huggingface.co is the way to go. Chances are high that someone else already did what you're interested in and it's a lot faster to download a pre quantized model.

Thebloke, turboderp, lonestriker, bartowski. Between these 4 users on huggingface you can find pretty much every common model already quantized.

What do you use to run them? My 3060 is generating just 40token per second with llama.cpp

I use either llama.cpp or exllamav2 as the backend, but speed is mostly dependent on how many layers fit in your gpu. I'm not sure which gpu I was thinking of for that 50-100 tok/s, on my 2070 I can only get around 15 tok/s on 7b models. I have to use 1.3b models to get to 100 tok/s on this gpu.

> Today’s large language models all adopt the Multi-head self-attention structure proposed in Google’s paper “Attention is all you need”. This is what people later call the Transformer structure.

Uhh... From the abstract:

> We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. —Vaswani, et al. 2017

This is interesting. While I think this approach may not be practical for a single computer due to poor performance, it could open new possibilities for distributed computing. Here, users across the internet could pool their graphics cards and the 80 layers could be distributed across 80 GPUs, with each GPU handling one layer. The results would then be processed through the system. Admittedly, if there is an average latency of 50 ms for each of the 80 GPUs, it would result in an additional 4 seconds of network delay. However, the overall process could be up to 80 times faster than on a single GPU since the layers are already in the GPU memory, albeit on different cards. This method also has the potential to enable the execution of much larger models on consumer hardware.

There is already an implementation along the same line using the torrent architecture.

https://petals.dev/

Does any one know the amount of data that is passed between layers? specifically the data size as this information determines the viability of distributed inference.

See https://github.com/FMInference/FlexGen

This is from much earlier this year and seems to be the same idea, only loading part of the model into the GPU at a time.

I read the whole article but did not find the words / token per second number?

An excellent article!

>"During inference, layers are executed sequentially. The output of the previous layer is the input to the next. Only one layer executes at a time.

Therefore, it is completely unnecessary to keep all layers in GPU memory. We can load whichever layer is needed from disk..."

...Or from the RAM of the main computer the Graphics Card is attached to, via the PCIe Bus and DMA (or whatever other methods exist now or in the future)...

>"...when executing that layer, do all the calculations, and then completely free the memory after."

...Or keep it in the adjacent RAM of the main computer, for more speed...

(Random idea here for hardware designers incidentally:

1) Graphics Card/GPU manufacturers -- put a high-speed CXL (Compute Express Link) interface on your Graphics card (https://en.wikipedia.org/wiki/Compute_Express_Link).

2) Independent Hardware Designers -- design a stand-alone RAM board (or RAM board rack/tower) which has a high-speed CXL interface on it, which can be interfaced to #1, above.

If #1 and #2 exist, now we have Graphics Cards/GPU's that can be interfaced to very large high-speed banks of external RAM -- effectively making the limited local memory of a GPU, a thing of the past...

Yes, there might be a slight latency, slightly above that of local GPU memory when accessing this adjacent memory. But I'm guessing that if this latency exists, then over time this will be engineered away...

In the short term, well designed software and/or caching strategies could compensate for any latency overhead, if there is one...

In conclusion, Graphics Card / GPU + CXL + External Rack o' RAM -- could be a very winning combination...)

>"This way, the GPU memory required per layer is only about the parameter size of one transformer layer, 1/80 of the full model, around 1.6GB."

A brilliant observation and a brilliantly written article!

Well done!

I did roughly the same thing in one of my hobby project https://github.com/fengwang/nnl. But in stead of using SSD, I load all the weights to the host memory, and while inferencing the model layer by layer, I asynchronously copy memory from global to shared memory in the hope of better performance. However, my approach is bounded by the PCI-E bandwidth.

Please explain, why GPUs cannot take weights directly from main memory? Why keep weights in VRAM? Why cannot I buy 256 Mb GPU and use it with main RAM?

Yes, NVIDIA GPUs can indeed read directly from main memory. It's called Heterogeneous Memory Management.

https://developer.nvidia.com/blog/simplifying-gpu-applicatio...

This would allow you to mmap() the weights file into CPU memory, and pass that CPU pointer to the GPU and allow it to fault the data in on-demand.

The GPU will have less bandwidth accessing this memory than if you copied it onto the GPU though, as it will need to take a trip over PCIe to read it. But it would eliminate the need to manually upload data to the GPU and the synchronisation involved with that.

Because GPUs only have VRAM on board.

For GPUs to take data from CPU RAM, it must traverse PCIe which is on the order of ~100x slower than through the on board VRAM.

Apple kinda does something like this with their GPU. They put it on the same chip as their CPU, added RAM very close to it with a lot of bandwidth and then use the RAM interchangeably between the CPU and GPU. Downside is that you can’t upgrade the RAM since it’s all stuck together in the same package as the CPU/GPU. It does allow you to get a CPU/GPU with 192GB of memory for pennies compared to something similar from Nvidia, but definitely not going to be as fast.

I don’t think the GPU can address all the memory though, I know that the 128GB is limited to 96GB that can be shared with the GPU.

Not only Apple does that. Intel and AMD have CPUs with integrated GPUs that do not have their own VRAM and use system memory. Why cannot they be used for LLMs? They don't cost an arm and a leg.

Intel Iris Xe can use half of the available memory. I don’t think it will help much though, the GPU is not very powerful and the bandwidth is around 90 GB/s.

The M2 Ultra in the Mac Studio has 800 GB/s. The Nvidia A40 has 696 GB/s while the Nvidia H100 SMX has 3.35 TB/s, which is probably the best you could get today.

https://www.intel.com/content/www/us/en/support/articles/000...

You can get two NVIDIA A40 cards with 48GB of memory each, second hand on eBay for 12 000 USD. You can connect them with NVLink to get 96GB effectively.

Or you could get a Mac Studio for 4 800 USD.

AMD (and Intel) has CPUs with integrated GPUs that use system memory. Cannot one use them as a reasonably priced alternative? I think that buying 64 Gb of ordinary memory is much cheaper than buying Apple's proprietary memory.

Because VRAM is faster than main memory, and a lot faster than ferrying data over PCIe.

Nvidia supports reading data directly from main memory. They introduced it around 2013 as Unified Memory, and it occasionally gets updates that make it more usable. It's probably faster than loading each layer into VRAM, computing it, discarding the weights and loading the next layer; but it's still a lot slower than having the weights in VRAM and often slower than just doing the work on the CPU.

It's called direct memory access. The CPU doesn't have to do all I/O. Any subsystem can access RAM with DMA. The main issue is scheduling the loads, stores, and arithemtic operations to avoid stalling the pipeline.

Wouldn't that be about as slow as CPU inference?

Probably much slower, due to all the memory thrashing and lack of pipelining optimizations.

lower end GPUs like T4

Alright, nothing to see here.

I could really use some kind of filter for stuff that does work with a beefy consumer GPU and stuff that does not.

We have tested this code on a 16GB Nvidia T4 GPU. The entire inference process uses less than 4GB GPU memory.

You don't need 16GB of VRAM for this, it's just what they had available for testing.

Hmm? A T4 is less powerful than a “beefy consumer gpu” like a 3080.

I'd say it is not the 4GB GPU setup that is the champion of sharding, but the 4xRTX3090 that is significantly cheaper than a single 80GB vram card.

Btw, has anyone experimented with Microsoft's DirectStorage api for this usecase? I understand that the shards will not be very compressable, but maybe the increased IO offloading can have benefits?

This is very clever. Now I'm wondering if someone has looked into optimal partitions and instruction streams. This would allow, in theory, to run any AI graph locally by partitioning the graph and loading it one piece at a time.

But anyway, if someone has $80B I can get this done. It's just an optimization problem.

Nvidea hates him: learn this one weird trick…

Can't wait for someone who will burn out an SSD drive by memory mapping it and running an LLM on it.

SSDs are only weared by writing AFAIK.

One Weird Trick, AI Edition

Please reword this clickbait headline. It's just disk offloading.

"Large language models require huge amounts of GPU memory. Is it possible to run inference on a single GPU? If so, what is the minimum GPU memory required?"

I don't like the cynicism here.

* Yes, performance numbers would be helpful.

* No, this is not new, and for most applications, not practical.

The basic point is that yesterday, I couldn't run a large model locally. Today, I can, in about one pip install and 20 lines of code. Yay!

There are a lot of things which are convenient to be able to test locally. YES it takes overnight, but if you're trying to confirm if there's a business case, that's often good enough. For a single compute (or even just a few), this is also simpler than spinning up a cloud machine.

You can essentially already do this with llama.cpp and ssd swap, much more quickly. Technically it only does the prompt processing and a few layers on the GPU (if that), but honestly that is better, just to avoid all the transfers over the GPU bus.

And you can do it in MLC, in your IGP, if you have enough CPU RAM to fit the model.

Running a 70B very slowly is nothing new. To be blunt, this strategy is a bad idea in the face of newer implementations.

Reply on news.ycombinator.com