SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup
hanlab.mit.edu
hanlab.mit.edu
It is important to keep in mind that modifying a model changes the performance of the resulting model, where performance is "correctness" or "quality" of output.
Just because the base model is very performant does not mean the smaller model is.
This means that another model that is the same size as the new quantized model may outperform the quantized model.
Suppose there are equal sized big models A and B with their smaller quantized variants a and b. A being a more performant model than B does not guarantee a being more performant than b.
Unlike traditional model compression work that simply applies existing techniques, SVDQuant synthesizes several ideas in a comprehensive new approach to model quantization:
- Developing a novel outlier absorption mechanism using low-rank decomposition — this aspect alone seems quite novel, although the math is admittedly way beyond my level
- Combining SVD with smoothing in a way that specifically addresses the unique challenges of diffusion models
- Creating an innovative kernel fusion technique (they call it “Nunchaku”) that makes the theoretical benefits practically realizable, because without this, the extra computation required to implement the above steps would simply slow the model back down to baseline
This isn't just incremental improvement - the paper achieves several breakthrough results:
- First successful 4-bit quantization of both weights AND activations for diffusion models
- 3.5x memory reduction for 12B parameter models while maintaining image quality
- 3.0x speedup over existing 4-bit weight-only quantization approaches
- Enables running 12B parameter models on consumer GPUs that previously couldn't handle them
And, I’ll add, as someone who has been following the diffusion space quite actively for the last two years, the amount of creativity that can be unleashed when models are accessible to people with consumer GPUs is nothing short of astonishing.
The authors took pains to validate their approach by testing it against three models (Flux, PixArt-Sigma, and SDXL) and along several quality-comparison axes (FID score, Image Reward, LPIPS, and PSNR). They also did a proper ablation study to see the contribution of each component in their approach to image quality.
What particularly excites me about this paper is not the ability to run a model that eats 22GB of VRAM in just 7GB. The exciting thing is the prospect of running a 60GB model in 20GB of VRAM. I’m not sure whether anyone has or is planning to train such a monster, but I suspect that Midjourney, OpenAI, and Google all have significantly larger models running in their infrastructure than what can be run on consumer hardware. The more dimensions you can throw at image and video generation, the better things get.
I am trying to call attention to the models used for evaluation comparison. There are 3 factors: inference speed/latency, model size in total loaded VRAM, and model performance in terms of output.
Comparisons should address all of these considerations, otherwise it's easy to hide deficiencies.
https://cdn.prod.website-files.com/64f4e81394e25710d22d042e/...
What I want to see is comparisons to NON-quantized models all with around the same VRAM along with associated inference latencies.
Also, we would want to see the same quantizing schemes applied to other base models.. because perhaps the paper's proposed quantizing scheme only beats others using a particular base model.
They also show it has little to no effect relative to fp16 on these models.
IMO that's enough. Comparison against smaller models is much less useful because you can't use the same random seeds. So you end up with a very objective "this is worse" based purely on aesthetic preferences of one person vs another. You already see this with Flux Schnell vs. the larger Flux models.
They report that their method produces a model that is 6.5 GB from flux (22.7GB). Why wouldn't you want to know how their 6.5GB model compares to other 6.5GB models?
Regarding aesthetic prefs: it's an open problem what an appropriate metric is for GenAI... LLM arena is widely regarded as a good way to measure LLMs and that's user preferences.
In any case, the authors report LPIPs etc. They could do the same for other small models.
But you're quite persistent in that they need to address this, so it seems much more likely they silently added it after your original post, or you didn't click through, concern trolling would stay more vague
isnt that the first image in the diagram / the 22GB model that took 111 seconds?
By showing their work side-by-side with other quantization schemes, you can also see a great example of the flavor of different results you can get with these slight tweaks (e.g., ViDiT INT8) and that their quantization does a much better job in reproducing the "original" (Figure 15).
In this application, it's not strictly true that you care to have the same results, but this work does a pretty good job of it.
Once a model has been trained, I believe the main metrics people care about are
1. inference speed
2. memory requirements
3. quality of output.
There are usually tradeoffs here. Generally you get a lower memory requirement (a good thing), sometimes faster inference (a good thing), but usually a lower quality of output.
I don't think reproduction of original output is the typical goal.
Take a look, it's good stuff! Basically a LoRA to reconstruct outliers lost by quantization, helping keep the performance of the original model.
E.g. compare quantized LLaMA 70B to unquantized LLaMA 8B.
Even better if the test model has a smaller version with similar byte size to the quantized larger one.
Its basically H100 speeds with 4090, 4.80it/s. 1.1 sec for flux schenll(4 steps) and 5.5 seconds for flux dev(25 steps). Compared to normal speeds(comfyui fp8 with "--fast" optimization") which is 3 seconds for schnell and 11.5 seconds for dev
nunchaku/third_party/spdlog/include/spdlog/common.h(144): error: namespace "std" has no member "function"
using err_handler = std::function<void(const std::string &err_msg)>;
^the conda always yes thing makes sure that you can just paste the script and it all works instead of having to press "y" for each install. Also if you don't feel like installing a wheel from random person on the internet, replace that step with "pip install -e ." as the repo suggests. I compiled that one with cuda 12.4 cause that was the part takes the most time and is what most often seems to be breaking.
Also I'm not sure if this will work on Fedora, I tried this on a runpod machine with 4090(apparently it only works on few cards, 3090, 4090, a100 etc) with Cuda 12.4 on host machine and "runpod/pytorch:2.4.0-py3.11-cuda12.4.1-devel-ubuntu22.04" this image as base.
EDIT: using pastebin instead as HN doesn't seem to jive with code blocks: https://pastebin.com/zK1z0UdM
[2024-11-09 19:33:55.214] [info] Initializing QuantizedFluxModel
[2024-11-09 19:33:55.359] [info] Loading weights from ~/.cache/huggingface/hub/models--mit-han-lab--svdquant-models/snapshots/d2a46e82a378ec70e3329a2219ac4331a444a999/svdq-int4-flux.1-schnell.safetensors
[2024-11-09 19:34:01.432] [warning] Unable to pin memory: invalid argument
[2024-11-09 19:34:02.143] [info] Done.
terminate called after throwing an instance of 'CUDAError'
what(): CUDA error: pointer does not correspond to a registered memory region (at /nunchaku/src/Serialization.cpp:32)but yea, can't help you outside of runpod, I haven't even tried this on my home PCs yet. for my usecase of serverless API, it seems to work
Its why fal, replicate, pretty much all big diffusion api providers use h100
tldr; 4090 is max 3.51 it/s even with all the current optimizations. h100 is 11.5it/s with all optimizations, and even without its 6.1 it/s
Paper discussing here is using 4 bit compute, which is 4x on 4090 in comparison with bf16 compute, while h100 doesn't have this at all (i.e. best you can get is 2x compute with fp8). So this paper will even out difference between those two to some extent. If to judge by theoretical numbers - H100 has 1979 TFLOPs fp8 compute, and 4090 has 1321 TOPS. Which puts it around ~65% of performance. Given the price of it ~$2K compared to H100s ~$30K this seems like a very good deal.
But again, no 4090 in DCs.
I had to int4 a llama model to get it to properly run on my 3060.
I'm curious, how much resolution / significant digits do we actually need for most genAI work? If you can draw a circle with 3.14, maybe it's good enough for fast and ubiquitous usage.
Which is rather unfortunate as it means that the difference between what you can train locally and what you can run locally is growing ever larger.
If you could train models faster, you’d be able to build larger, more powerful models that outperform the competition.
The fact that Llama 3 is significantly over trained than what was considered ideal even three years ago shows there's a strong appetite for efficient training. The lack of progress isn’t due to a lack of effort. No one has managed to do this yet because no one has figured out how.
I built 1-trit quantized models as a side project nearly a decade ago. Back then, no one cared because models weren’t yet using all available memory, and on devices where memory was fully utilized, compute power was the limiting factor. I spend much longer trying to figure out how to get 1-trit training to work and I never could. Of all the papers and people in the field I've talked to, no one else has either.
What did you try? What were the research directions at the time?
It's missing the colourful commentary that I'd usually give, but alas, we can't have it all.
Before that, in 2012, Alexnet had to be partially split into two submodels, running on two GPUs (using a form of interlayer grouped convolutions) because it could not fit in 3GB of a single 580 card.
Ternary networks appeared in 2016. Unless you mean you actually tried to train in ternary precision - clearly not possible with any gradient based optimization methods.
> The number of model parameters is slightly higher in the BitLinear setting, as we both have 1.58-bit weights as well as the 16-bit shadow weights. However, this fact does not change the number of trainable/optimized parameters in practice.
https://arxiv.org/abs/1603.05279
I personally have a pretty negative opinion of the bitnet paper.
>we introduce BitLinear as a drop-in replacement of the the nn.Linear layer in order to train 1-bit weights from scratch
>While the weights and the activations are quantized to low precision, the gradients and the optimizer states are stored in high precision to ensure training stability and accuracy. Following the previous work [ LSL+21 ], we maintain a latent weight in a high-precision format for the learnable parameters to accumulate the parameter updates. The latent weights are binarized on the fly during the forward pass and never used for the inference process.
https://arxiv.org/pdf/2310.11453
The other paper had a much nicer and clearer introduction to bitlinear than the original Microsoft paper, which is why I used it. Uncharitably you might say that they aren't hiding the lead 10 paragraphs in.
Necessary precision depends on, unsurprisingly, what you're truncating. Flux drops off around q6. Text generation around q4.
The llms apple are putting in iphones are q4 3b models.
Diffusion models, however, are computationally bound, even for single batches, so quantizing weights alone yields limited gains.Quantizing in general helps with the memory bottleneck but does not help in reducing computational costs, so it’s not as useful for improving performance of diffusion models, that’s what it’s saying.
This means that the only saving you're getting in speed for a diffusion model is being able to do more effective flops since the floats are smaller, e.g. instead of doing one 32bit multiplication, you're doing 8 4bit ones.
By comparison for transformers you not only gain the flop increase, but also the improvement in memory shuffling that they do, e.g. it also takes you 8 times less time to load the memory into working memory from vram.
The above is a vast over simplification and in practice will have more asterisks than you can shake a stick at.
> To achieve measured speedups, both weights and activations must be quantized to the same bit width; otherwise, the lower precision is upcast during computation, negating any performance benefits.
tries to explain that.
What it means though is that if you only store the inputs in lower precision, but still upcast to say bf16 or fp32 to perform the operation, you're not getting any computational speedup. In fact, you're paying for upconverting and then downconverting afterwards.
You won't however have access to usual sampler, latent image, Lora nodes to do anything beyond basic t2i
This is the whole magic, the rest of the workflow doesn't need to unload and flush memory, causing big delays for jobs.