Hacker News Reader

Comment by vimy | Hacker News Reader

> That being said, such a split design may help with improved AI inferencing.

Unified memory is the only reason Macs are so coveted right now for local AI. A single 192 gb ram Mac costs less than the equivalent in standalone GPUs.

View on HN

The execution speed for LLM inference gets so slow once you reach models that even fill out a 64GB machine. I was tempted, too, but then realized it was unusable past 48GB-ish and stacking used 3090s was the best price / perf / vram ratio.

What are the good use cases for very large memory amounts?

Mixture of Expert models, where all parameters must be in memory but only a subset are accessed per token, are a sweet spot for Macs.

DeepSeek v3 for instance has 671B params, but should have the memory bandwidth of a 37B dense model with a batch size of one.

Any model you'd recommend for a Mac with 48 GB RAM?

48GB is maybe just enough to squeeze a quantized 70B model in, like Llama 3.3, but you'll need to raise the GPU memory allocation limit [1] and it might not be super fast.

You could also try Qwen 2.5 32b, which you should just work with ollama or LM Studio with no config changes.

I've got a 32gb M1 Max and a 24gb 4090, and I barely ever run models on my Mac, as the memory bandwidth and compute for prefill is much better on the 4090. But I'm essentially locked out of Llama 3 70B class models, which I only use via API.

[1] See: https://www.reddit.com/r/LocalLLaMA/comments/186phti/m1m2m3_...

It's slower than using just GPU RAM, but it's still faster than using a regular PC that has a much more limited bandwith between the main memory and the GPU. It's some sort of middle ground on how fast and how cheap you can do inference with LLMs that don't fit into a consumer GPU's RAM.

It's definitely usable past 48GB, I have a 96GB M2 Max and regularly run models that use around 70GB that are very usable.

I also have a home server with 2x3090 and 2xA4000 (80GB vRAM) - yes it's a lot faster, but it's a pain in the ass to build, it takes up a lot of space, uses 10x the power, and honestly - cost about the same as my MacBook Pro.

I ran all my research papers (about LLMs) on a Mac Studio.

Unusable in what way? Llama 3.3 70b q8 with 100k+ context runs as well as GPT 3.5 did a couple years back, except local on Mac and smarter.

Large context has different perf than large model. Op was likely thinking of running 400b models and finding the compute wasn’t enough to make the memory useful.

Look on youtube running various models and even a 70b model does a slow 3tok/s

If you don't have tensor parallelism even 8 x A100 is going to struggle with that kind of model.

This isn't splitting the unified memory, it's splitting the soc into a sip. The ram is still on the same interposer/substrate. The actual mm distance won't be regressing to say the distance of a dimm socket to the cpu socket.

The software will still see a single memory pool

I know im disagreeing with the article

I'm sorry, I think you mispronounced market segmentation... This is Apple we're talking about.

When LLMs and diffusion models began spreading freely, it was very funny to have support for some beefy high end GPUs and MacBook Air M1.

I hope Apple sticks with the architecture. Even if its not very practical, its great to have it as possible.

I don’t think the principal market actually cares about that.

Yeah, this is the edge Macs have right now in the AI space. It's why people are looking forward to the Strix Halo from AMD as it will also have a unified memory architecture and will probably cost a good bit less than a Mac.

Reply on news.ycombinator.com