Hacker News Reader

Comment by reissbaker | Hacker News Reader

Apple's take on GPUs is quite different and very interesting IMO. Shared memory architecture with absolutely massive RAM (and hence VRAM) support, e.g. the new Mac Studio having 192GB RAM/VRAM which can run pretty massive models, much more so than is easily consumer-accessible even at the high end with Nvidia 4090s. It's not as fast as Nvidia, but it's not horribly far off in the latest chips.

As LLM adoption grows, I wonder whether Apple's approach will start to make more sense for consumer adoption, so that you can run models on your machine without needing to pay large subscription costs for AI-powered apps (since OpenAI et al have fairly high fees). The high cost of using the APIs in my opinion is a drag on certain types of adoption in consumer apps.

Llama.cpp actually first started as a way to run LLMs on Macs! At first CPU-only, but then later the first GPU driver backend added was Metal, not anything from Nvidia.

View on HN

I agree, though it’s still vendor lock-in. I think a lot of devs aren’t aware of how tightly integrated these frameworks are with the hardware. It’s not trivial to separate the two and the companies driving this tech have no motive to do so, while also being amongst a very small number of hardware designers.

I think criticizing Apple for not supporting Nvidia is a classic missing-the-forest-for-the-trees. Unifying memory is a next logical step in general purpose computing. The flexibility that comes from it has unseen potential.

Thinking what Apple sells is a GPU bundled with a large amount of VRAM and a free ARM CPU makes things easier to accept.

NVIDIA never sold a GPU with expandable memory, either.

It's a next step that AMD has been trying to make happen for like 10 years, but not managing to make it commercially successful in anything but consoles, ending up permanently stuck in the lowest low end of the market. In both the PC and datacenter market spaces it has ended up with basically the opposite of product market fit. Nobody actually wants the generic CPU compute tied to the GPU compute.

The PC gamers really want the GPU component to be separately upgradable from the CPU. Non-gamer PC users don't care about the GPU performance, just cost. The datacenter folks want to be able to use a single $1k CPU to host $100k worth of GPUs.

It's plausible that AI accelerators follow a different path for the consumers. It's harder to see it happening for the datacenter market.

You’re missing the forest for the trees.

Being able to switch from an optimized CPU-centric workload to an optimized GPU-centric workload without any hardware changes sounds useful to me.

You could even do unified memory with upgradeable separate CPU and GPU. You won’t get the benefits of having them on the same chip, but there’s nothing intrinsic about the separation requiring separate memory space.

Is that RAM as fast as A100/H100 VRAM? AFAIR, gpus push ~ 1 TB/s ish

The Apple M2 Ultra memory bandwidth is 800GB/s so it's not a long way off 1TB/s.

A100 goes up to 2TB/s in the largest configuration, and H100 claims 3TB/s per GPU node. (These figures keep changing with new variants.) But you can buy several Mac Studios for the price :-)

The real use of H100 is for training, as it can pool many GPUs together with a dedicated high bandwidth network. You can't do that with Mac Studios.

That figures are really impressive. A single node is still useful for finetuning smaller models. Apple could move the needle in this market a bit

In addition to the cost, OpenAI censors their models, and while that seems protective at first, if you think about it, if the model is the knowledge graph, then censor data in the graph is censoring free speech.

I don't personally like model censorship, but you need to come up with a better argument than...whatever that is. It is entirely unconvincing, and seems like you misunderstand what free speech is.

Freedom of speech is a principle that supports the right of an individual or a community to articulate their opinions and ideas without fear of retaliation, censorship, or legal sanction.

Since the models are not suppressed to answer certain questions and indeed have been demonstrated to be biased toward one end of the political spectrum, and if you treat the model’s output as a knowledge graph, as proposed by John Schulman, one of the cofounders of OpenAI, then yes, I would say the suppression of freedom of speech is a valid argument to make. Otherwise why would there be a set of “uncensored” models that exist in the open source world?

I would suggest you read about these models and think about the implications. Perhaps that’ll lead to you to reconsider your stance

If we ignore the corporate structure details where your analogy breaks down, in the simplest case, choosing to self-inhibit is not violation of the freedom of speech. OpenAI is voluntarily self-censoring their output. This isn’t an ideological battle for them, it’s a business.

> I would suggest you read about these models and think about the implications.

I have done that, and come to the conclusion that "that's business, baby!"

Do you have any more meaningful objections to this "censorship" of a free thing you agreed to license?

Reply on news.ycombinator.com