Hacker News Reader

Comment by RcouF1uZ4gsC | Hacker News Reader

> For example, M2 Ultra can train massive machine learning workloads in a single system that the most powerful discrete GPU can’t even process.

It's interesting to see that Apple is explicitly targeting the M2 Ultra as an alternative to GPUs for machine learning.

I think it is great for consumers for Nvidia to have a competitor in this space. If you wanted to do ML, you really didn't have much of an option other than Nvidia, which allowed Nvidia to do things like disable certain features in consumer GPUs even though the hardware was capable and limit RAM. Having Apple as a competitor in the consumer space will force them to up their game.

View on HN

That unified memory architecture makes them well suited in many cases - that's the one bit current x86 desktop platforms don't have a good answer for, where the system design generally keeps GPU and system memory as entirely separate technologies.

This is why fanless M2 MacBook Airs are regularly loading models 5k dollar PCs might choke on - you can get 24gb of unified memory, while almost all high end consumer GPUs top out at 16gb.

High end consumer GPUs would be your 3090/4090 cards which both have 24GB.

4080 is still a 16gb part, and still the 2nd fastest part Nvidia sell at consumer level (its a 1200 dollar part). The 3090 (for however long it remains on sale) and the 4090 are outliers.

The obvious counter point is an M2 MacBook in 14/16 inch size, where wallet permitting you can have up to 96gb of unified memory too...

I simply chose the Air in my example earlier because its incredible you can have that level of performance in ostensibly an entry level passively cooled machine.

You don't have that level of performance though. Not even close really. RAM amount is only one piece of the puzzle.

Well at least in the U.S. a lot of used 3090s are out for less than $800 on eBay which is a steal for 24GB VRAM. The 4090 is about $1600 and it's still a consumer-level card; gamers are buying it more often than the 4080. For ML workloads, it's actually a pretty good deal considering how expensive Nvidia's higher-end, non-consumer offerings are. Lambda Labs did some benchmarking: https://lambdalabs.com/blog/nvidia-rtx-4090-vs-rtx-3090-deep...

But yeah the M2 MacBooks are incredible for local LLMs for their price. Nvidia doesn't have any consumer-level priced accelerators with that much memory.

I don't know is 24GB Air qualify "entry level"

A 4090 is $1500-2000 depending on the vendor. A M2 MacBook Air with 24GB RAM is in the same price range, and it's a whole-ass laptop.

Don't AMD APUs have unified memory?

Not with the same bandwidth.

Yeah but M2 effective speed for ML is closer to intel CPUs than it is to nvidia gpus. The correct comparison here is to compare how fast you can train on Apple M2 vs an Intel CPU with equivalent RAM.

Yup!

> It's interesting to see that Apple is explicitly targeting the M2 Ultra as an alternative to GPUs for machine learning.

For many purposes the limiting factor on local LLMs is the amount of VRAM. nVidias with 64GB+ are insanely expensive (unless you're a funded startup). The ability of Apple Sillicon GPUs to use system RAM for the GPUs is a game-changer.

> The ability of Apple Sillicon GPUs to use system RAM for the GPUs is a game-changer.

It'd be a lot more of a game change if Apple didn't continue its tradition of charging extortionate prices for ram.

While I agree it's expensive it's not quite apples to apples. GPU memory is very very expensive. A100 80GB costs $10k msrp (> $15k on ebay). An RTX 4090 only has 24gb of vram at $1600. Even if you do dual 3090's + nvlink for $1600 you only get up to 48GB, no where near the up to 192GB of vram you can access on apple silicon. The upgrade from 64GB to 192GB of ram is $1600. That's still expensive of course, but 128GB of 800GB/s memory that your GPU has access to for $1600 is actually not bad value. It's not like just sticking an extra 128gb of ram into your computer. DDR5-6400 dual channel will give up up to 102 GB/s, which is no where near M2 Ultra or GDDR6X.

Reply on news.ycombinator.com