Hacker News Reader

Comment by 1-6 | Hacker News Reader

Question is, how much of an improvement has it gotten to over a GPU or ASIC?

So... I was struggling with this for a while. I would says anywhere from 2x to an order of magnitude faster with a GPU. (I've been looking at a lot of GPU benchmarks lately, and they are REALLY hard to compare since they are all so specific)

I do think long term there gets to be more hope for CPUs here with inference largely because memory bandwidth becomes more important than the gpu. You can see this with reports of the MI-300 series outperforming h100, largely because it has more memory bandwidth. MCR dimms give you close to 2x the exiting memory bw in intel cpus, and when coupled with AMX you may be able to exceed v100 and might touch a100 performance levels.

HBM and the general GPU architecture gives it a huge memory advantage, especially with the chip to chip interface. Even adding HBM to a CPU, you are likely to find the CPU is unable to use the memory bw effectively unless it was specifically designed to use it. Then you'd still likely have limited performance with things like UPI being a really ugly bottleneck between CPUs.

If someone releases DDR5 or DDR6 based PIM, then most of the memory bandwidth advantage of GPUs evaporates overnight. I expect CPUs to be king at inference in the future.

But then you'll get GDDR6 delivered via HBM5 or whatever. I don't think CPUs will ever really keep up with the memory bandwidth, because for most applications it doesn't matter.

MCR DIMM is like 1/2 the memory bandwidth that is possible with HBM4, plus it requires you to buy something like 2TB of memory. It might get there, but I'd keep my money on hbm and gpus.

I think that should be phrased more like "what fraction of GPU speed can this reach?", because it'll always be less than 1x.

Nothing in software will ever beat an equivalent ASIC.

Sure there is. Software is easy to change.

By “beat” I meant in performance.

Obviously you can’t change an asic

an asic is fixed function, so it'll never be able to boot my pc and then be the CPU, even though an asic beats the pants off anything else computing Sha hashes for Bitcoin mining.

By “beat” I meant performance.

Obviously an ASIC is not a general purpose machine like a cpu.

Most ASICs are cost or power optimizations.

Exactly. They’re much faster for their specific tasks and thus are more power efficient and potentially cost efficient

No. Eg of the hardware discussed on the article, the Raspberry Pi uses an ASIC that's slow, cheap and low power vs the Intel or AMD chips.

In some cases ASICs are faster than general purpouse CPUs, but usually not.

I think we’re saying the same thing here.

You can make an ASIC which doesn’t have the same power draw as a CPU, but provides the same performance.

It doesn’t need to be faster than the fastest software implementation, but power per performance will always favor ASIC.

Is the LLM running on an ASIC for the Pi here? I dout it.

The ARM cores and the VideoCore parts are all in the ASIC, it's a SoC type ASIC.

Hmm, yeah it's a SoC, but not an ASIC. Maybe you mean APU? ASICs are circuits thst can only do one thing, CPU cores are definitely not that.

Edit: an example ASIC the Pi has is the video encoder/decoder, with JPEG also supported. I think it's embedded in the GPU.

ASIC just means it's an application specific IC (= chip), meaning it's fabbed for that specific product (like in this case the Raspberry Pi). A functional block like a JPEG codec contained therein is not an ASIC. Quoth wikipedia:

"Modern ASICs often include entire microprocessors, memory blocks including ROM, RAM, EEPROM, flash memory and other large building blocks. Such an ASIC is often termed a SoC (system-on-chip)."

What you're describing in the JPEG codec might be termed a fixed function IP block in semiconductor design terminology.

First paragraph of that Wikipedia article:

An application-specific integrated circuit (ASIC /ˈeɪsɪk/) is an integrated circuit (IC) chip customized for a particular use, rather than intended for general-purpose use, such as a chip designed to run in a digital voice recorder or a high-efficiency video codec.

That’s weird that Wikipedia says there’s no real distinction between SoCs and ASICS.

Colloquially, I’d never call anything with a programmable processor an ASIC

Some ASICs are SoCs. Some are not.

From the article, passage about the 14900k:

> For example, when I run my spam.sh shell script, it only takes 420 milliseconds, which is 7x faster than my Raspberry Pi 5. That's right, when it comes to small workloads, this chip is able to finish before CUDA even gets started.

So… it depends :)

I think I understand what you are thinking. You may be fixing "than other ways of running them" to the end of the title, but it's actually "than it was on CPU before now".

Reply on news.ycombinator.com