$95 AMD CPU Becomes 16GB GPU to Run AI Software | Hacker News Reader

$95 AMD CPU Becomes 16GB GPU to Run AI Software

tomshardware.com

185 pointsnateb202262 comments

> one minute and 50 seconds to generate a 512 x 512-pixel image with the default setting of 50 steps

2s/iter? That is... Not great.

It is slower than CPU diffusion: https://github.com/rupeshs/fastsdcpu

Stable Diffusion in particular doesn't need much VRAM anyway. I get that many people are stuck on lower end computers, but ~4GB is not a totally unreasonable requirement.

Looks more like 4 s/iter which is actually quite good for CPU inference in Automatic111. It is an apples-to-oranges comparison to the LCM models used by fastsdcpu (since those can get away with fewer sampling steps) or to the quantized models favored by apple (since they use totally different underlying compute capabilities). Currently automatic111 and other popular GUIs are pretty focused on users with beefy GPUs that can sample images in seconds, so they haven't yet implemented some of these really significant CPU-friendly speedups. But we'll get there!

Are those cpu speedups out there? Where can I find them?

Not for A1111's SAI backend, as far as I know.

VoltaML has some CPU optimized options for the torch.compile optimization.

That's probably exactly what you would expect from a GPU running off DDR4 instead off GDDR6X.

> Stable Diffusion in particular doesn't need much VRAM anyway. I get that many people are stuck on lower end computers, but ~4GB is not a totally unreasonable requirement

Wow, did it really get that low? I played with SD during the initial days, and I know there was some strides then to get it running on 8gb cards and maybe squeaking through on 6gb with a smaller image size (like, 256x256px).

Can you generate reasonably large images on 4gb now, with just a higher wait time? I was never sure why it wasn't simple to just shuffle bits in and out of main RAM but I guess the model themselves are mostly big binary blobs that are hard to split up?

> Wow, did it really get that low?

For SD 1.x and even 2.x, yes, in A1111 I ran both on a 4GB card before upgrading to 16GB 3080Ti laptop, and not even with --lowvram; reportedly, --lowvram will let you do at least 1.x on a 2GB card at normal (512x512) resolution directly.

> Can you generate reasonably large images on 4gb now, with just a higher wait time?

Yes, especially with tiled diffusion/multidiffusion.

>Wow, did it really get that low?

Always has been. I can generate 768x768 with 3.9GB used, and I don't even use --lowvram or --medvram options in Automatic1111. But note that if you don't have an Intel CPU with onboard video, you will need to lower your resolution because because your OS is using a bunch of VRAM for the desktop already, more than the 0.1GB you left free.

Yeah, I’m not sure where that figure came from either. My experience on an 8GB card is that it is always nudging up close to the limit and you don’t want to try to turn the knobs up much past the default.

SDXL is a total nonstarter as well on 8GB.

Using vladmandic sd.next with v1-5 works fine* on my RX 580 4GB at around 1s/it on ROCm and 1.5s/it on DirectML.

fine = requires --medvram or --lowvram in order to not run out of memory. The speeds above are with medvram, which is just about on the edge of what's possible. These options compromise speed but are still better than running on the CPU generally.

VoltaML is also quite good on low vram cards.

Does VoltaML have any support for AMD/Intel GPUs or CPU inference? e.g. I think Intel's openVINO supports ONNX, but the Github page only mentions Nvidia and CUDA.

The docs are somewhat out of date, and TBH I'm not sure about openVINO I know some are running it with ROCm.

> Can you generate reasonably large images on 4gb now, with just a higher wait time?

You are stuck at low res, but you can use a GAN to upscale the output.

SDXL needs a lot more VRAM however, especially when training LoRAs

It's also significantly slower than Apple M1, which is 2.2 iteration per second in iPad. Though, maybe it's not a fair comparison since Apple optimises heavily their library.

[0] https://github.com/apple/ml-stable-diffusion

This is similar to the Apple GPU trick where unified memory means it's easier to load large models into GPU space, but as others have said, the AMD APU built into these chips is _really_ underpowered, even compared to other RDNA APUs like the one in the Steam Deck

Minor nitpick but the 4600G and 5600G mentioned in the article are Vega, not RDNA. The iGPUs in those two processors are/were quite powerful by iGPU standards at the time, especially considering they were competing with Intel HD graphics or very early Iris Xe iGPUs. Not a scratch on a good dedicated GPU though.

Steam Deck is RDNA2 though, and is significantly faster. I don't think RDNA1 ever made it into an APU.

If you switch to the test-fp8 branch of AUTOMATIC1111 web UI, you can do 512 * 1024 on 4G GPU with SDXL.

(74 days ago )

"AMD Ryzen APU turned into a 16GB VRAM GPU and it can run Stable Diffusion"

https://news.ycombinator.com/item?id=37162762

I've just finished writing an article on hardware required for LLM's in production https://bionic-gpt.com/blog/llm-hardware/

Would be interesting to see how this performed running Llama 2 7b.

Last month I did testing on the latest gen AMD APU (7940HS @ 65W - 8C Zen4 CPU, Radeon 780M RDNA3 GPU) on how llama2-7b (4-bit quants) performed w/ a variety of inferencers (llama.cpp, exllama v2, mlc w/ CPU, CLBlast, ROCm, and Vulkan backends) and only llama.cpp's ROCm implementation convincingly beating CPU inferencing.

Even then, interactive (batch=1) inferencing was still barely faster. I suspect memory bandwidth (I ran w/ 2 x DDR5-5600) is the main bottleneck. Just as a point of reference, the inferencing performance is roughly in the ballpark of (but slightly lower than) my 16GB M2 MBA.

Details: https://docs.google.com/spreadsheets/d/1kT4or6b0Fedd-W_jMwYp...

Yes. Without batching, every weight is only used once per token. On discrete GPUs, you generally need around ~20 flops/byte of memory bandwidth to be compute bound. I don't know where APUs are in that regard, but probably not entirely different. Memory bandwidth divided by model size is more or less the token rate you should expect to get from a good LLM inference implementation.

2 channels of DDR5-5600 = 89.6GB/s theoretical max bandwidth and a Q4_0 quant takes about 6.7GB of VRAM used, which is 13.4X, so that maps out pretty well for inferencing speed (I think the divided by parameters maybe only works for 4bit in this case). Interestingly, the ROCm implementation does go a bit faster (>15 t/s so it must have some tricks up its sleeve).

The Radeon 780M does 8.9 TFLOPS of FP32, 17.8 TFLOPS of FP16, so if 20 flops/byte applies, it has roughly 10X more compute than memory bandwidth for inferencing.

A few things:

You're a bit apostrophe happy. Sure, as a matter of style and preference you can use apostrophes to pluralize short words / acronyms, but "it's" is a contraction for "it is".

An AMD Ryzen 2700X system for $1500? That might've been a reasonable price in 2018, but you couldn't reasonably spend that kind of money nowadays if you tried.

I'd love to see some more numbers showing implementations that specifically target certain platforms. For instance, you can run LLaMa on a Raspberry Pi, if you really wanted, with certain compromises. How would a fleet of small, inexpensive systems compare with a CPU-based and GPU-based systems?

It's interesting, all the same :)

> "it's" is a contraction for "it is".

It's true, but it seems orthographically inconsistent with other possessive forms; after all, "John's" can be a contraction for "John is," or it can be the possessive form of the name "John." By analogy, one might think that "it's" can also take on both meanings, but that's not the case.

At this point the M3 Max in a MacBook Pro with 96/128GB V/RAM needs to be benched against various Nvidia cards like 4090 & the H100 for $/performance numbers

The sad thing is all of this hardware is outrageously overpriced.

Its even sadder that AMD/Intel have failed to take advantage of the price gouging. They could have released affordable 32GB-48GB cards with (afaik) no silicon changes.

The 7900 XTX is still somewhat reasonable where you can get it to work.

True, Intel is trying its best at both hardware & recently software contributions like Neural Compressor, so let's see how far they take nvidia to task

Well, they never released a 32GB A770. This is particularly perplexing, as they have no professional market to lose like AMD does.

And the iGPUs are still pretty small.

We will see how Meteor Lake and Battlemage shake out, but I don't like the delay rumors.

If I remember correctly, Intel isn't making a lot of money on these GPUs. Their main goal for now is gaming, and you don't need 32GB cards for that.

If they intend to release an AI card, I bet they'll charge through the nose for it, just like Nvidia does.

Well, thats also unfortunate, as its in Intel's/AMD's interest to push into the local AI market with low pricing first.

I just thought of this and the mobile chips with DDR would even work and probably be super affordable (not as fast) or even GDDR. Not sure how intel packages their iGPU and if it can do this as well - I have only followed the AMD side. I suppose nVidia could snag a lower end card and modify the memory interface and do the same thing quite easily using an older fab node.

They can't swap out GDDR for DDR.

But what they can do is double up the GDDR chips (which is what the 48GB pro cards do), kinda like running 2 DIMMS per channel on desktops. This requires a somewhat more expensive PCB and support from the memory controller, but AMD already supports this for sure, and I would be shocked if the Intel A770 die can't do it.

No, you need GDDR memory bandwidth for machine learning. Also GDDR ICs are dirt cheap. GPUs with lots of VRAM are only expensive because of price discrimination.

You might be disappointed when it comes to Stable Diffusion on M series it’s very slow compared to even 3090.

Impressive it works at all but the difference is night and day compared to top of the line (gamer card) Nvidia. Like SDXL <7 seconds a generation vs 60+ seconds.

Why the H100? Comparing against similarly priced consumer gear (3x 4090) makes a lot more sense, but I think you already know that one of these systems has an order of magnitude better perf/$ than the other

Yeah even that as well

Not sure if you missed my point. 3x 4090 will be 10x faster while costing the same. There's no point making the comparison.

(Assuming both machines are running the same model. I see a lotta crazy numbers thrown around based on comparing very different quantizations)

INT4 or FP4 ones are of interest to me

Running LLama 2 70b a Mac Studio gets 13 tokens per second.

2 x 4090 is more like 38 tokens per second.

But it's harder than that. As the 2 x 4090 is using https://llm.mlc.ai/ which is highly optimized so perhaps the Mac Studio inference could be better when tuned.

Then a lot of production inference engines use continous batching, which means they perform really well when processing multiple prompts at a time.

What quantization are you talking about for these?

You can squeeze 70B onto a single 4090/3090 with exllamav2.

As of the latest 8 bit cache commit, I can manage 2.6bpw with seemingly good quality.

For inference or training?

For training, Apple has no CUDA, so you're pretty much dead in the water, unless you're specifically interested in e.g. local fine-tuning (for e.g. privacy or start-up strategy of "use the user's compute").

For inference, 4090 and H100 are not designed for throughput per $. Nvidia GPUs like the A40 or T40 are tuned for cloud inference, but at the end of the day your product is either running inference server-side (so play your own CAPEX vs OPEX, maybe even try Jetson) or user-side and so the user chooses the hardware not you.

I mean maybe there are Wallstreet analysts or people at Qualcomm/Broadcom who want an Apple vs Nvidia shootout to see who is getting the most of their silicon, but for the product space the comparison is basically irrelevant. Which is likely what Apple and Nvidia want right now.

It's slow, badly support and doesn't scale like gaming. It's useless.

Did similar with a 4000 series APU.

It’s a bit weak for LLMs under near any circumstances but thinking it could probably handle two whisper cpp decodes over opencl and maybe some embeddings api stuff.

The author seems to "define" GPU as "what can run A.I. workload". Well: despite the fact that modern GPUs have indeed become more CPU-like, in that they are now suitable for GP (general-purpose) computing, it does not make sense to call a CPU a GPU, even out of good intentions - because it's confusing; a GPU still only plays the role of a co-processors, and its single-thread performance is low compared to the CPU's, among other limitations.

So I'm a bit out my field here but I've been wondering...

Is there something that prevents AI software from augmenting VRAM with system RAM? My impression was that for instance OpenGL doesn't even have a standard way to get VRAM usage because you're not supposed to care, stuff is just shuffled around in the background for you if VRAM runs out.

It's not very useful because the point of VRAM is that it's very very high-bandwidth. CPU RAM is slow and the PCIe link to the GPU is even slower.

What's the usage pattern like? Is all of the VRAM used extensively in AI workloads, or one could hope to augment things a bit with system RAM with little performance impact?

The moment you can't fit the model in VRAM and have to use PCIE the performance impact is MASSIVE.

Depending on the type of model/usecase/how much you can fit in VRAM it very well may still be usable, however compared to fitting everything in vram it'll be abysmal for most any model type.

From my understanding the entire model must be read per iteration (or very close to all of it. This is why offloading any part of it impacts the performance so much.

Diffusion models depending on the sampler and some other stuff can potentially get away with only a few steps (usually 2-20 best case) and are relatively small (3B or less).

However LLM's are generally much bigger (they mostly start at 7B, there's a few exceptions depending on the use case), and they need an entire iteration per token which is only a few characters (think 2 to 3). So llm's especially are far more useful if you can keep them in vram.

Thanks, I messed around with this stuff a little bit, but have little understanding of what goes on in the guts of it.

The performance impact is not little it's huge.

My SDXL workflow was often spilling over my RTX 3080’s 10GB of VRAM into system RAM. I’ve since tweaked my workflow to prevent that, as the slowdowns are dramatic. Tiled VAE encode/decode was a big help.

> Is there something that prevents AI software from augmenting VRAM with system RAM?

As I understand, it adds complication, the points where it is practical without just getting into an horrible churn of shuffling are limited based on model architecture, and (as a consequence) implementations may not always been transferrable between models outside of a narrow family.

On the other hand, there are some frameworks with general support which eases this.

On the third hand, shuffling stuff back and forth between system RAM and VRAM has a performance cost in the best case.

So, yes, and no, and yes.

Most (that I've seen) consumer-focused frontends for local models (text generation, image generation, audio generation) do it, to a greater or lesser extent.

The base model implementations that are getting thrown out by researchers don't usually focus on that, and its generally implemented by third parties after the fact, sometimes with commonalities that can be applied to other models that are closely-enough related.

The latest nvidia drivers do this automatically, but it's very slow.

lol "non-serious tasks" what are those?

Hm. Every task that isn't directly or indirectly going into a production system? Any task where reliability, performance (including performance per $) and quality don't matter?

This includes hobby projects, tinkering out of curiosity, tasks performed for learning purposes (not necessarily educational) and more.

Look, my hobby is edge ML projects using SOC hardware. They are hacky AF but still more serious than some hack CPU approach like this. If anything, the next malware is turning unknowing teenager gamers rigs into piles of slag thanks to this useful discovery.

I guess this article just rubs my net negative lifestyle and feels like its content for contents sake.

I read this twice and have no idea what you're trying to say.

Just saying you can do better for less and with less effort.

Misleading title and article. The CPU doesn't "become" a GPU, it already shippes with one. What the engineer did is solely use the Ryzen APU's GPU for ML.

Reply on news.ycombinator.com