Hacker News Reader

Comment by microtonal | Hacker News Reader

Maybe, but most of these comparisons are based on GPU performance (ie. games). For other workloads like machine learning, Tensor cores on NVIDIA GPUs will blow Apple Silicon GPUs out of the water. The M2 Ultra is 27 TFLOPS. The 4080't Tensor Cores are 48.7 TFLOPS in FP32, 194.9 TFLOPS in FP16, 389.9 TFLOPS in FP8 with FP16 accumulate. (IIRC on Apple Silicon GPUs FP16 performance is roughly the same as FP32.)

(There is the Neural Engine which supports lower precision, but it limited in various ways.)

Regardless, the strides that Apple has been making are impressive.

View on HN

I think the exciting thing with Apple silicone and "AI" is inference, not training. Due to their unified memory, you can potentially have enormous local model inference. That's potentially much more expensive on a PC with a GPU having its own memory.

Apple have an opportunity, if they 2-4x the memory on the entry level devices (not beyond the realms of possibility), to make local inference a thing available to all.

Sure, but for inference it’s much easier to work in lower precision. Training with very low precision is also possible, but is often more tricky due to numerical stability.

A lot of work is going on in 8-bit inference and even 4 bit inference. So, models that need 64 GB in FP32 can do with 16GB VRAM in FP8 or INT8, which is well within the realm of consumer NVIDIA cards. And the latest NVIDIA tensor cores will absolutely destroy Apple Silicon GPUs or the Neural Engine in 8 bit.

So, I don’t think it’s really a strong argument. And as someone who is a Mac user and a ML practitioner, I’d be very happy if they started supporting eGPUs again.

Apple Silicon has many strengths and the GPU core are fine for many ends, from games to graphics apps.

But let’s not pretend that Apple is beating NVIDIA at their own game (yet). That day might come, but currently it only leads to disappointed users in ML forums who were hyped into thinking that their vanilla M2 MacBook Airs can almost compete with a 4090 in training a deep transformer model. (Yes, that happens.)

Multiple NVIDIA GPUs also have unified memory from an application perspective, one GPU has up to 80GBs right now. They also have a proper software stack. If NVIDIA saw any credible threat there, they could ramp that up, DRAM is not beholden to Apple.

One differentiator is the amount of memory available. Apple's shared pool of memory means the GPU has potentially up to 96gb of ram at its disposal, where discrete GPU's are often limited to 8, 12, 16, etc. They mentioned in the keynote that the M2 can handle many tasks discrete GPUs cannot simply because they run out of ram.

$15,000 will buy you an A100 with 80GB of RAM at 2TB/s that will annihilate the biggest Apple chip.

Sure, but then that’s the price of multiple entire laptops. Or 2 Mac Pros. Not sure what point you’re trying to make here.

It is also the performance of 4 Mac Pros. GP was talking about tasks that dedicated GPUs supposedly could not handle. Who is is running models needing 60+ GB of VRAM on their laptops?

Some have had fun running some of the larger Llama, Vicuna & Guanaco models on their MacBooks. Mostly to experiment with local inference.

Although quantisation has now lowered the required memory, I wouldn’t be surprised if it comes in handy again in the near future.

But today that’s quite a niche use case.

FLOPS are one thing, but 192 GB of unified memory that can be used as VRAM is something else. That could be a big win on the inference side of things, where even an RTX 4090 GPU is limited to only 24 GB.

Then there's the power consumption difference to consider. This seems like one of those cases where benchmarks reveal only a fraction of the larger picture.

On purpose to keep a100 and h100 on an additional premium.

Reply on news.ycombinator.com