Hacker News Reader

Comment by moneycantbuy | Hacker News Reader

Anyone know if high-RAM Apple silicon such as the 128 GB M1 Ultra is useful for training large models? RAM seems like the limiting factor in DL, so I'm hoping apple can put some pressure on nvidia to offer consumer GPUs with more than 24GB RAM.

View on HN

Having large amounts of unified memory is useful for training large models, but there are two problems with using Apple Silicon for model training:

1. Apple doesn't provide public interfaces to the ANE or the AMX modules, making it difficult to fully leverage the hardware, even from PyTorch's MPS runtime or Apple's "Tensorflow for macOS" package

2. Even if we could fully leverage the onboard hardware, even the M1 Ultra isn't going to outpace top-of-the-line consumer-grade Nvidia GPUs (3090Ti/4090) because it doesn't have the compute.

The upcoming Mac Pro is rumored to be preserving the configurability of a workstation [1]. If that's the case, then there's a (slim) chance we might see Apple Silicon GPUs in the future, which would then potentially make it possible to build an Apple Silicon Machine that could compete with an Nvidia workstation.

At the end of the day, Nvidia is winning the software war with CUDA. There are far, far more resources which enable software developers to write compute intensive code which runs on Nvidia hardware than any other GPU compute ecosystem out there.

Apple's Metal API, Intel's SYCL, and AMD's ROCm/HIP platform are closing the gap each year, but their success in the ML space is dependent upon how many people they can peel away from the Nvidia Hegemony.

1:https://www.bloomberg.com/news/newsletters/2022-12-18/when-w...

This basically seems to line up with what another tech community (outside of AI / ML) seem to agree on: that Apple hardware is not that great for 3D modeling or animation work. Their M1 and M2 chips rank terribly on the open-source Blender benchmark: https://opendata.blender.org/

Despite Apple engineers contributing hardware support to Blender: https://9to5mac.com/2022/03/09/blender-3-1-update-adds-metal...

So looks like NVidia (and AMD to some extent) are winning the 3D usage war as well for now.

They're pretty darn good for laptop GPUs.

This isn't really an Apple specific problem - GPUs with low power budgets are never going to beat GPUs intended to permanently run off of wall power.

(I know there's stuff like the Mac Studio and iMac, but those are effectively still just laptop guts in a (compact) desktop form factor rather than a system designed from the ground up for high performance workstation type use)

I'd love to see a dedicated PCIe GPU developed by Apple with their knowledge from designing the m1/m2, but it's not really their style.

Specifically, a Macbook Air M2 ranks more or less like a NVidia GTX 970 of 2014.

But... the GTX by itself has a 145W TDP, whereas the whole laptop has a TDP of 20W!

This is super impressive imho, but of course not in the raw power department.

The 4090 is able to draw 450 watts. Current apple silicon are all used on laptops, which cannot possibly sustain or cool that much power. It might change when they make a cooling-optimized desktop chip though.

Not exactly an answer to your question, but from a pytorch standpoint, there are still many operations that are not supported on MPS[1]. I can't recall a circumstance where an architecture I wanted to train was fully supported on MPS, so some of the training ends up happening in CPU at that point.

1 = https://github.com/pytorch/pytorch/issues/77764

I have a MacStudio Ultra with 64 GPU cores and 128 GB of RAM which I use it to play around with neural networks (among other things). I can create models larger than what I would be able to on consumer GPU cards but it isn't very fast. On average, I get about 20% of the speed of an RTX 3090.

It is definitely useful but does not beat the top Nvidia cards. [1]

Would be very interesting to retest with the M2 and also in the months/years to come as the software reaches the level of optimisations we see on the PC side.

[1] https://wandb.ai/tcapelle/apple_m1_pro/reports/Deep-Learning...

It's an interesting option. My gut instinct is that if you need need 128GB of memory for a giant model, but you don't need much compute - like fine tuning a very large model maybe - you might as well just use a consumer high core CPU and wait 10x as long.

5950X CPU ($500) with 128GB of memory ($400).

CPU-addressable RAM is not interchangeable with graphics card RAM, so unfortunately this strategy isn't quite in the right direction.

AFAIK it flat out won't work with the DL frameworks.

If I'm mistaken please do speak up.

Edit: Thank you JonathanFly, I didn't know this!

I think they are saying train (or at least fine-tune) on a CPU.

This can work in some cases (years ago I certainly did it for CNNs - was slow but you are fine tuning so anything is an improvement) but I don't know how viable it would be on a transformer.

All the frameworks work on CPU. At the time I tried it, the 5950X was about 10x slower than my GPU, which was a 1080Ti or 2080Ti. GAN not a transformer though.

If you are training a model that requires this much memory, it will also require a lot of compute, so it would be too slow and not cost-effective. It may be useful for inference.

in a word .... no

Reply on news.ycombinator.com