For people wanting to do LLM work at home, a fully loaded MacBook has more memory available to it than anything else a consumer can buy. All in a laptop form factor (versus Nvidia GPUs).
A small market, sure, but one they have on lock down.
A small market, sure, but one they have on lock down.
I think the Jobs/Ive legacy that has pushed the thermals ahead of raw compute had given them a massive headstart in the LLM age for both notebooks (obviously) but also desktop (less obviously).
With the size of a normal 4090 heat sink the ATX platform is showing its age. The biggest element of any system is a series of huge heat sinks. The graphics card ends up being massive. You have support struts coming out. Giant power supplies. Bespoke connectors. Paying for RAM twice.
Apple have a form factor people are comfortable with and haven’t set the bar too high with raw performance. They aren’t hindered by existing designs. The thermals are good.
And you pay for it all in one.
Desktop CPUs using DDR5 have low memory bandwidth because they have narrow bus widths (128bit total) at low speeds. Server CPUs use the same memory at the same per-pin speeds but with much larger bus widths to achieve total bandwidth comparable to a mid-range GPU but supporting orders of magnitude more memory capacity. Apple's high-end chips likewise uses significantly larger bus widths than desktop CPUs, combined with slightly per-pin speeds (LPDDR rather than DDR) to also reach GPU-like total bandwidth.
I don't think it's accurate to single out any one of the above technologies as being the "normal" DRAM. GDDR, LPDDR, and DDR are all fairly different from each other and all quite mainstream.
LPDDR is always at least as close to the processor, and is often in-package so significantly closer. HBM is always in-package.
But all of that is a red herring, because the distance is only loosely correlated with performance and is a completely irrelevant metric if you've already accounted for bus width and clock rate.
Are you talking about SoCs? Regular laptop/desktop APUs don't usually include their own RAM.
Not really. It's pretty difficult to do LLM development work on a smartphone.
It would appear the poster misread reports about the L2 cache (96MB) for the VRAM (24GB).
Sure the m3 with 128GB of RAM costs $7k, which is an absurd number, but that will run high quality LLMs that can have sci-fi level intelligent conversations.
Not a bad price for a laptop capable of doing things that was literally "haha that'll never happen" levels of unimaginable 5 years ago.
That would be more than an H100, which costs around $30k.
1. https://www.tomshardware.com/news/nvidia-makes-1000-profit-o...
That's literally what BOM means: bill of materials. It's not supposed to include any of those other costs. BOM is not always the right kind of cost to be talking about, but in a case like this where you're talking about just swapping out the memory on an existing product, BOM accounts for all the costs that change significantly.
The blog post really highlights how effective those headlines are at getting clicks but not informing readers: it leads with “nearly 1,000%” because that sounds even more rage bait-y than the already high “823%”, and that's clearly the impression people take away from the story even though starting with the second paragraph it makes it clear that this is a very fuzzy claim which is likely significant understating the true cost for Nvidia to make the hardware as sold or the other factors I also mentioned.
Finally, “swapping out the memory on an existing product” can mean “we replaced part A with larger part B” or it can mean “we upgraded a controller, added additional channels, and redesigned the physical form factor to provide better mechanical and thermal capacity for the extra chips”. That's why those extra details are so important since it's otherwise easy for someone to make a tweet which sounds like they know what they're talking about but is in fact leaving out a lot of important context.
Not very portable either.