FlexGen: Running large language models on a single GPU
github.com
github.com
For those that aren't familiar with it they are tiny power and density optimized GPUs. I have the successor (A2) and total max TDP is 60 watts. Single slot. Slot only power, and passively cooled.
Depending on workload I observe it to be roughly 5-10x slower than a 3090, which means for most people at home with a spare Nvidia gaming card (or whatever) you'll see results from these project at a performance multiple of the benchmarks they provide.
The one caveat is that the T4/A2 have 16GB VRAM, which makes them more capable (albeit slower) than a "low end" desktop card like the 3070 which has only 8GB VRAM. But as HN readers know there is incredible progress daily to reduce VRAM requirements for these models!
I'm finding that my 12GB 3060 is not enough to run even LLaMA-7B without CPU offload, thank Arceus for quantization...
That's the only thing that worked for me, but it was so easy and worked instantly.
It's a set of easily-auditable batch/bash scripts (depending on platform).
Over the years I've made several attempts at AMD/ROCm. I've wasted so much time AMD owes me free GPUs for life ;).
At this point I've just accepted that AMD is irrelevant in ML and they're fine with that - they're not making much of an effort to change that. I really and truly wish that wasn't the case but I've accepted that it is.
Moving to a lower level, check out the release notes for the latest Nvidia driver and look in amazement at the sheer number of supported GPUs[2]. In short, literally every GPU they've put in a laptop, desktop, workstation, or datacenter over the past decade (plus their embedded Jetson stuff). There are over 2,000 GPUs listed there and I can tell you from experience the "Unified" in CUDA holds up. If the card is supported by the driver, it has a compatible compute arch, and enough VRAM whatever you throw at it will just work. In many cases even on Windows! People complain about the proprietary driver but the fact is if your GPU says Nvidia on it you can install the one driver on any distro and have these projects up and running in minutes.
Compare that to the ROCm "list"[3] of what, maybe a dozen GPUs? I know from experience (unfortunately) that even with "supported" hardware in many cases just getting the driver to work is a nightmare. Then you have basic frameworks randomly crashing, etc. It's a complete mess and like I said - AMD is a decade behind today and Nvidia's lead is only growing.
[0] - https://docs.nvidia.com/deeplearning/frameworks/support-matr...
[1] - https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorc...
[2] - https://download.nvidia.com/XFree86/Linux-x86_64/530.41.03/R...
[3] - https://docs.amd.com/bundle/Hardware_and_Software_Reference_...
Enabling ROCm support for unlisted AMD GPUs & APUs is a feature flag away if your running one of these janky enterprise distros that AMD has blessed.
Debian is slowly getting ROCm and PyTorch's packages into the main Debian archive, once complete using AMD hardware for machine learning should be a breezey Apt Install ROCm away, as Debian's machine learning team chooses sane defaults like enabling APU support by default. Derivative distros like PopOS, Linux Mint, Zorin, Kali, Ubuntu and such will inherit this easy support from upstreaming ROCm as well.
When I last tried it a year or so ago it was (more or less) useless. Yes it “ran” but there were weird crashes and edge cases all over the place. Throw in the fact that 95% of documentation, examples, tools, benchmarks, etc are still for CUDA after four years or so of embarking on this journey I think I’ve finally given up for good.
I happily and very firmly live in Nvidia/CUDA land now where I can see a story on HN and have it running on my GPU in under 10 minutes. Or 30 seconds if it’s a docker container.
You’re brave for running on unsupported GPUs! My experience was bad enough with the “supported” ones :).
I'm thiiiiis close to throwing up my hands and getting a 4070 but I've been advised that other than ML/CUDA, AMD>Nvidia for linux. Also the 4070 is 2x the price for just slightly better performance.
There is so much interesting, educational, and productive stuff going on with GPUs these days… At some point people need to ask themselves “did I buy a GPU to learn/use ML, or did I buy a GPU to beta test AMDs software stack?”
When it comes to GPU compute Nvidia cards are substantially cheaper if you value your time and sanity.
Note I don’t have any vested interest in Nvidia whatsoever, I’m just mad at myself for spending my time and money thinking AMD actually cared about any of this - they clearly don’t and the lack of progress I’ve seen and experienced over the past five years I’ve tried with them is insulting to me and the rest of their so called user base.
Same here. AMD's neglect of non-gaming uses is likely how Intel is going to become number 2 in GPUs.
I'm all for global cooperation and fellowship. Are sanctions going to be a barrier for this and related projects?
I could not find any specific information on how these sanctions affect academic research and cooperation between the US and Russia. Some sources suggest that some academics have canceled conferences, joint projects, and funding with Russian institutions as a form of self-imposed sanctions2, while others indicate that Russian students are still able to secure visas to study abroad3. Therefore, the impact of the sanctions on academic research and cooperation may vary depending on the field, institution, and individual circumstances.- The best sanctions impact targeted industries (e.g. anything needed to build tanks, warplanes, etc.).
- The worst sanctions impact communications and collaboration. Change comes from conversations. Media, non-military education, and non-military academic collaboration are bad usually bad sanctions.
There's a lot of effort on finetuning these large models on < 80GB VRAM hardware but significantly more effort on "running" them on < 24GB VRAM hardware (3090/4090 and lower).
In the end your point stands - between any efforts by those impacted by sanctions or just the hobbyist/low-mid range commercial applications research into cramming these huge models into "consumer" hardware are accelerating dramatically.
Also it's likely good part of their research staff already work from Israel and Serbia even though they not finished splitting the company into Russian one controlled by Kremlin and international one.
PS: As I see only contributor in repository from Yandex is located in Hungary and not in Russia.
Anything new with the codebase?