Hacker News Reader

Comment by jsheard | Hacker News Reader

16GB base RAM across the board, following the iMac. AI is certainly good for pushing up the baseline RAM that manufacturers can get away with shipping if nothing else.

View on HN

Also the base ram for the pro chip is 24gb. I hope it will be the same for the MacBook Pro.

Unified memory too. It's your GPU and your ram.

This is huge for AI / ML at least for inference. Apple chips are among the most efficient out there for that sort of thing, the only downside is the lack of cuda

> Apple chips are among the most efficient out there for that sort of thing

Not really? Apple is efficient because they ship moderately large GPUs manufactured on TSMC hardware. Their NPU hardware is more or less entirely ignored and their GPUs are using the same shader-based compute that Intel and AMD rely on. It's not efficient because Apple does anything different with their hardware like Nvidia does, it's efficient because they're simply using denser silicon than most opponents.

Apple does make efficient chips, but AI is so much of an afterthought that I wouldn't consider them any more efficient than Intel or AMD.

For inference, Apple chips are great due to a high memory bandwidth. Mac Studio is a popular choice in the local Llama community for this particular reason. It's a cost effective option if you need a lot of memory plus a high bandwidth. The downside is poor training performance and Metal being a less polished software stack compared to CUDA.

I wonder if a little cluster of Mac Minis is a good option for running concurrent LLM agents, or a single Mac Studio is still preferable?

The memory bandwidth on Apple silicon is only sometimes comparable to, and in many cases worse than, that of a GPU. For example, an nVidia RTX 4060 Ti 16GB GPU (not a high-end card by any means) has memory bandwidth of 288GiB/sec, which is more than double that of the M4 CPU.

On the higher end, building a machine with 6 to 8 24GB GPUs such as RTX 3090s would be comparable in cost (as well as available memory) to a high-end Mac Studio, and would be at least an order of magnitude faster at inference. Yes, it's going to use an order of magnitude more power as well, but what you probably should care about here is W/token which is in the same ballpark.

Apple silicon is a reasonable solution for inference only if you need the most amount of memory possible, you don't care about absolute performance, and you're unwilling to deal with a multi-GPU setup.

Note that they said the _Mac Studio_ which in the M2 model has between 400GB/s and 800GB/s memory bandwidth.

https://www.apple.com/mac-studio/specs/

Edit: since my reply you have edited your comment to mention the Studio, but the fact remains that the M2 Max has at least ~40% greater bandwidth than the number you quoted as an example.

Yeah, sorry, I realized that as well so I edited my post to add a higher end example with multiple 3090s or similar cards. A single 3090 has just under 1TiB/sec of memory bandwidth.

One more edit: I'd also like to point out that memory bandwidth is important, but not sufficient for fast inference. My entire point here is that Apple silicon does have high memory bandwidth for sure, but for inference it's very much held back by the relative slowness of the GPU compared with dedicated nVidia/AMD cards.

It's still "fast enough" for even 120b models in practice, and you don't need to muck around with building a multi-GPU rig (and figuring out how to e.g. cool it properly).

It's definitely not what you'd want for your data center, but for home tinkering it has a very clear niche.

> It's still "fast enough" for even 120b models in practice

Is it? This is very subjective. The Mac Studio would not be "fast enough" for me on even a 70b model, not necessarily because its output is slow, but because the prompt evaluation speed is quite bad. See [0] for example numbers; on Llama 3 70B at Q4_K_M quantization, it takes an M2 Ultra with 192GB about 8.5 seconds just to evaluate a 1024-token prompt. A machine with 6 3090s (which would likely come in cheaper than the Mac Studio) is over 6 times faster at prompt parsing.

A 120b model is likely going to be something like 1.5-2x slower at prompt evaluation, rendering it pretty much unusable (again, for me).

[0] https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inferen...

Exactly, the M2 Ultra is competitive for local inference use cases given the 800 GB/s bandwith and a relatively low cost and energy efficiency.

The M4 Pro in the Mini has a bandwidth of 273 GB/s, which is probably less appealing. But I wonder how it'd compare cost-wise and performance-wise, with several Minis in a little cluster, each running a small LLM and exchanging messages. This could be interesting for a local agent architecture.

See my sibling reply below, but I disagree with your main point here. M2 Ultra is only competitive for very specific use cases, it does not really cost less than a much higher-performing setup, and if what you care about is true efficiency (meaning, W/token, or how much energy does the computer use to produce a given response), a multi-GPU setup and Mac Studios are on about equal footing.

For reference comparing to what the big companies use, an H100 has over 3TB/s bandwidth. A nice home lab might be built around 4090s — two years old at this point — which have about 1TB/s.

Apple's chips have the advantage of being able to be specced out with tons of RAM, but performance isn't going to be in the same ballpark of even fairly old Nvidia chips.

The cheapest 4090 is EUR 110 less than a complete 32GB RAM M2 max Mac Studio where I live. Speccing out a full Intel 14700K computer (avoiding the expensive 14900) with 32 GB RAM, NVMe storage, case, power supply, motherboard, 10G Ethernet … and we are approaching the cost of the 64GB M2 ultra which has a more comparable memory bandwidth to the Nvidia card, but with more than twice the RAM available to the GPU.

Apple will let you buy more RAM for cheaper than Nvidia, but it won't be the same speed — it'll be ~20% slower than a 4090.

That's my point. I would absolutely be willing to suffer a 20% memory bandwidth penalty if it means I can put 200% more data in the memory buffer to begin with. Not having to page in and out of disk storage quickly make those 20% irrelevant.

If you have enough 4090s, you don't need to page in and out of disk: everything stays in VRAM and is fast. But it's true that if you just want it to work, and you don't need the fastest perf, Apple is cheaper!

How _exactly_ do I keep 50+ Gigabytes of data in the 4090's VRAM without paging back and forth to disk?

As the person you’re replying to said, by having multiple 4090s. 3090s work pretty well also, and are less than half the cost of 4090s.

How is that relevant when the discussion from the start was about comparing a two year old Mac with a two year old GPU as a cost-benefit discussion.

In any case how are you going to fit 50+GB in two (theoretically 24+24 GB) Nvidia cards without swapping to disk when the Mac in question has 64GB (also theoretically) available?

You seem confused. Please feel free to read my post near the top of this very chain of comments, where I specifically compare a Mac Studio to a machine with 6 to 8 Nvidia GPUs. That was the discussion “from the start.”

> In any case how are you going to fit 50+GB in two (theoretically 24+24 GB) Nvidia cards

No one was talking about only two cards.

> Mac Studio to a machine with 6 to 8 Nvidia GPUs.

That post read like a joke comparison and still does. Can you elaborate how it is relevant?

What seems like a joke about it? And relevant to what, exactly?

The parent of my initial comment in this thread said: "For inference, Apple chips are great due to a high memory bandwidth... It's a cost effective option if you need a lot of memory plus a high bandwidth."

My post was attempting to explain at a high level how 1) Apple SoCs do not really have high memory bandwidth compared to a cluster of GPUs, and 2) you can actually build that cluster of GPUs for the same cost or cheaper than a loaded Mac Studio, and it will drastically outperform the Mac.

If you want specifics on how to build such a GPU cluster, you can search for "ROMED8-2T 3090" for some examples.

I hope this helps.

And yet the GPU costs about as much as the whole Mac Mini and wouldn't even come close to fitting inside one.

You're mostly correct, though a 4060Ti 16GB is 20-30% cheaper than the cheapest Mac Mini. More importantly though, "fits inside a Mac Mini" is not a criterion I'm using to evaluate whether a particular solution is suitable for LLM inference. If it is for you, that's fine, but we have vastly different priorities.

Does it use the GPU? I was under the impression that it uses the CPU. It's only faster because of the massive memory bandwidth compared to DDR4/5

The AI features use all three of NPU ("ANE"), GPU, CPU, mostly depending on model size.

https://machinelearning.apple.com/research/neural-engine-tra...

Frankly you’re very wrong. NPUs and GPUs aside, 16gb of GPU memory is very rare in consumer hardware

You can't use all 16GB because it's unified, so it's shared with the system, SSD controller etc. You can use something like 12-14GB though.

Sure, still incredibly rare in a $600 device

I'm not sure what you mean. RTX 4060 Ti/4070 Ti Super/3090/4090 cards can be easily purchased at any major electronics store in person or online and have 16GB or 24GB depending on model. Once you get up to 32GB, your point would stand, but 16-24GB GPUs are common.

Yes but the average user is not purchasing those, let alone putting together a system with one for $600

You said nothing about price in your initial comment, and the cards I listed are some of the most popular GPUs of the last several years.

Lack of Cuda is not a problem if for most ML frameworks. For example, in PyTorch you just tell it to use the “mps” (metal performance shaders) device instead of the “cuda” device.

Cuda Apple license it from nVidia?

Cude, er, cute, but... no.

I tried training some models using tensorflow-metal a year ago and I was quite disappointed. Using a relu activation function led to very poor accuracy [0] and training time was an order of magnitude slower than just using the free tier of Google Colab

[0] https://github.com/keras-team/tf-keras/issues/140

That simply isn't true in practice. Maybe for inference, but even then you're running up against common CUDA kernels such as FlashAttention which will be far from plug and play with PyTorch.

I consider that a plus. Maybe the AI community wil start to wake up and realize that going all in on cuda is ridiculously stupid.

Not if you're an NVDA shareholder!

To be totally honest, there's enough money in the ML / AI / LLM space now that I fully expect some companies to put forward alternative cards specifically for that purpose. Why google does not sell their TPU to consumer and datacenter instead of just letting you rent is beyond me.

Ceiling is too low on alternative hardware for that.

Until Apple can bang something out as good as an h100, it's no competition.

Cuda thrives bc of the hardware offering too.

So, do you think that when the Mac Studio gets upgraded, it will also come with less max RAM, but be unified?

Is the whole "unified" RAM a reason that the iMac and Mini are capped at 32G?

The Mac Studio has always had unified memory

All "Apple Silicon" products, going back to the first one, which was the iPhone 4.

Fun fact: any PC with integrated graphics has also had unified memory (yep, including Intel Macs), for at least the past decade!

True, but PCs are 128 bits wide, apple lets you upgrade to 256 bits wide (M4 pro), 512 bits wide (m3 Max) or 1024 bits wide (M2 Ultra).

Unified memory is much more useful when you can get more bandwidth to it.

Yep, there's no performance x86 CPUs on the market with ambitious GPUs, only laptop chips. Games are optimized for discrete GPUs, Apple didn't have that software inertia to deal with.

Sort of, obviously quite a few games are optimized for the PS5 and Xbox series X.

GPU cores are generally identical between the iGPUs and the discrete GPUs. Adding a PCIe bus (high latency and low bandwidth) and having a separate memory pool doesn't create new opportunities for optimization.

On the other hand having unified memory creates optimization opportunities, but even just making memcpy a noop can be useful as well.

GPUs are all about (compute & memory) bandwidth. Using the same building blocks of compute units doesn't yet it go fast. You need a lot of compute units and a lot of bandwidth to feed them.

The performance dependency on DGPUs doesn't come from the existance of a PCIe bus and partitioned memory, but from the fact that the software running on the DGPU is written for a system with high bandwidth memory like GDDR6X or HBM. It creates opportunities for optimization the same way as hardware properties and machine balances tend to, the software gets written, benchmarked and optimized against hardware with certain kinds of performance properties and constraints (like here compute/bandwidth balance and memory capacity, and whether CPU & GPU have shared memory).

Yeah the same tech pcs were using for 14+ years

https://x.com/LinaAsahi/status/1820947147312820497

> I know ancient iGPUs had that thing for setting the GPU memory size in the BIOS, but that's aaaaaancient and completely obsolete. If you still have that, just set it to the minimum value. The rest of memory will be unified.

I hadn’t used a PC in so long, I still thought that bios setting decided the division. TIL.

Lucky we have Asahi Lina to clarify the details.

It shows up as "Shared GPU memory" in Task Manager. What BIOS sets is the Dedicated i.e. reserved video memory in RAM.

e.g. My Ryzen iGPU reserves 2GB/32GB for itself (which Windows can't see) via BIOS and use 9 more as shared "unified" memory.

The on-chip RAM means that you can run models on the CPU that would require the GPU on a peecee.

The RAM is not on the chip. I need to get a tee shirt and a bumper sticker that says this.

The reality distortion field is not dead, rebranding the iGPU like this even convincing the technical crowd has been a great marketing win for Apple.

No mention of the SSD size that was the reason I returned my Mac mini 256Gb last year was just a pain juggling files

> No mention of the SSD size

The base model is 256 gb. You can see it here:

https://www.apple.com/shop/buy-mac/mac-mini

Base model is 256GB configurable up to 2TB, all others start at 512GB, the M4 Pro model can go up to 8TB

It's $800 to go up to 2TB from the 256GB model which is just criminally over priced. I can get double that for half the price with a Gen 4 NVMe drive. Weirdly the 8TB drive on the Pro is at least in line with the top of the line 8TB NVMe SSDs you can buy though there are cheaper options at about $600 vs the $1200 Apple is charging.

That's regular Apple pricing for you. Great deals on the baseline models, but insane margins on the upgrades that make them usable. And of course the ability to upgrade the devices yourself has been phased out in the name of performance and power efficiency

in the name of [share price] performance and [market] power efficiency

Edit: sorry, answered to the wrong post.

I just checked some Dell prices: $730 to upgrade an XPS desktop from 512GB to 4TB (Apple charges $1200), or $508 to upgrade an Optiplex tower from 256GB to 2TB QLC, or $654 to upgrade it from 256GB to 2TB TLC (Apple charges $800). Scalping on upgrade pricing is something all the PC OEMs do.

Yeah, but the 5 minute job of installing a cheap retail SSD in that Dell machine yourself is still an option which Apple has removed from all but the Mac Pro, which offsets any SSD savings by being $3000 more expensive than an equivalently specced Mac Studio.

Right. The repairability argument is the reasonable discussion to have. The silly pricing games are more of a red herring.

I think it's more relevant with Apple because they've removed all the competition for basically all upgrades to their devices by either soldering things to the board or bundling them into their SOC. When there's no alternative their prices become the only option.

The Mac Studio technically still has socketed SSDs, which presumably cuts costs by not having to manage a separate motherboard SKU for every SSD capacity, but they went out of their way to design a proprietary SSD module format rather than just using the standard...

Kind of, they were socketed NAND cards and the controller lived on the mainboard, so as far as getting out of the problem of Apple entirely setting the prices for everything it's not relevant. As far as I'm finding no one managed to find a way to create a compatible card to create an avenue for DIY upgrades. I've found a few upgrades but they consist of buying an entire second Mac Studio to harvest the drive from.

Giles from Polysoft is manufacturing 3rd party mac studio nand cards. There is still a problem sourcing nand because apple doesn't let the owm sell to anyone but them.

I thought I had seen something about that but couldn't find the actual boards mentioned for sale. Sounds even worse though because it should be possible but Apple being Apple has ensured there's no source for Giles or other companies to perform repairs.

There's an industry in China for desoldering and reusing apple bands. Unfortunately getting new oem nands is going to take legislation.

Yes, though that relies on the product being fairly popular and for the chip to be stable for a while for it to be a useful source. Mac Studio NAND chips aren't going to be readily available from them unless they happen to be a shared part from a more popular device.

All macs use the same compatible nand chips.

Is there any compatibility between generations, eg. harvesting NAND from a M1 generation machine to upgrade a M3 (or now M4) generation machine?

As far as I'm aware, yes they're all compatible.

Apples segmentation takes place in the nand firmware. The firmware contains the location in the storage configuration. And this may or may not be rewriteable. Iboffrc has done a video explaining how some of it works. It's all from reverse engineering though.

I think repairable or not, the pricing to upgrade to the max config just isn't something a price-sensitive consumer should take seriously. Beyond one or two upgrades, you might as well pretend it says "call for quote" and just not consider 4+ TB as a realistic option to get from the OEM, because those prices are trying to cause sticker shock. And that goes for any PC OEM—the price-gouged upgrades are so far beyond reasonable that it really doesn't matter whose prices are the most silly or by exactly how many hundreds of dollars. What does matter is whether aftermarket upgrades and repairs are possible.

It matters that there are no after market options for Apple because it means the inflated OEM upgrade price is the ONLY price available for every given upgrade. It matters less with Dell/Asus/Lenovo etc. because that's not the only price available.

The top of the line is also not where Apple is gouging the worst. It's in the middle tiers that are actually relevant to many more people. Most don't have a need for 4+ TB main drives but 1-2 TB is a size that's pretty easy to justify for a lot of people and Apple's price is the only option for them and they're absolutely lining their pockets with cash at the expense of anyone not going for the bargain bin basic tier that can't hold 2 modern games.

By all means, complain about the cost to get a 1TB config, and put that price in proper context. But it still doesn't make much sense to focus on the $1200 upgrades, or any of the other upgrades whose price rounds up to "lol, no".

The gouging is bad at all levels as well as the design effects on repairability and the issues with obsolescence. If you'll look way back though I acknowledged the $1200 is at least vaguely in line with the top of the line Sabrent 8tb NVMe of the same size.

"Repairability" is a red-herring when the discussion is about user-upgrades and the ability to purchase components from 3rd-party suppliers (who compete against each other and the OEM)

Apple is getting at throwing this red herring.

At least on the storage side, Apple's parts are neither more performant nor all that much more power-efficient than a standard, replaceable SSD.

Apple's silicon is good but I don't see what's so special about all the other stuff. Looks like they just solder components to the motherboard instead of using industry standard interfaces.

*industry standard physical interfaces. The electrical interace is bog standard.

Phased out in the name of profit efficiency. Despite what marketing will tell you the SSD is industry standard NVMe and the RAM is standard LPDDR5.

> I can get double that for half the price with a Gen 4 NVMe drive

It's worse than that -- 4TB gen 4 drives can be had for well under $300, sometimes $225-250, and that's for buying a drive outright, not "trading up" from a 256GB device. I think it'd be more accurate to say that you can get double the capacity for a _quarter_ of the price.

I was ballparking it based on my recent buy of a Samsung 990 Pro 4TB and inflated the price a little in my head to closer to $400 than the $330 it actually was.

I also, as a side note, try to give the loosest most favorable (to my opposite) comparison because when I err on my side it becomes a "well actually" debate a lot of the time about how it's "not quite X times as many it's more like X-1 (so I'm not even going to touch that X-1 is still quite bad)" that is really tedious and annoying especially when the favorable version of the comparison is still quite bad for their point/side.

Very much so. When I bought my "cheesegrater" Mac Pro, I wanted 8TB of SSD.

Except Apple wanted $3,000 for 7TB of SSD (considering the sticker price came with a baseline of 1TB).

I bought a 4xM.2 card and 4x2TB Samsung Pro SSDs, cost me $1,300, I got to keep the 1TB "system" SSD, and was faster, at 6.8GBps versus the system drive at 5.5.

Similar with memory. OWC literally sells the same memory as Apple (same manufacturer, same specifications. Apple also wanted $3,000 for 160GB of memory (going from 32 to 192). I paid $1,000.

It's USD2,400 to upgrade the M4 Pro model from 512GB to 8TB, which feels a bit steep, but it's an option.

Alternatively, you can get one of these[1] external Other World Computing NVME SSDs for USD1,190 right now. And then you can easily move all your files from your laptop to your desktop when you get home.

[1] https://eshop.macsales.com/item/OWC/US4EXP1MT08/ (15% off list price as of writing)

You can get some decent size external usb SSDs. I have the Samsung T5 2TB. I think they have larger models now. Works pretty well. And with USB-C speeds are very usable. You can probably get faster/bigger stuff via thunderbolt.

I'm considering getting one and a nice big monitor or TV. It needs to run x-plane 12 at decent speeds and maybe support a bit of light gaming. My macbook M1 pro is actually pretty decent for this but the screen is too small for me to easily read the instruments. I expect this will do better even in the base setup.

Otherwise my needs are pretty modest. I'd love to see steam add some emulation support for these things as I have some older games that I enjoy playing. I currently play those on a crappy old intel laptop running linux. I've also been eyeing a new AMD mini PC with the latest amd stuff (Beelink's SER9).

Seems pretty nice as well and seems like it is more performance for the money. Apple is doing its usual thing of charging you hundreds of euros for 50 euro upgrades. Get the base mac studio instead. It probably makes more sense if you are going down that path.

The big problem is that there are lots of stuff that macOS won’t let you move to a external drive, like iCloud Drive.

It's a desktop. Use an external disk. Much cheaper.

It kind of undermines the sleek form factor if you need to have a clunky NVMe enclosure dangling off the back though. Even with this tiny new design I bet they could fit a hatch on the bottom with space for a 2230 M.2 drive, but they don't want to because that would let you upgrade to 2TB of fast internal storage for $200 instead of $800.

There are a number of companies that specialize in making hubs/NVMe enclosures that match the aesthetics of the Mac Mini and sit directly underneath it.

For example: https://www.amazon.com/dp/B08S47KBMC/

It looks like the power button was moved to the bottom of the new Mac mini, maybe to make this experience worse.

> clunky NVMe enclosure

We really are living in the future if people are using these words in combination.

Though compared to this new mini a lot will feel clunky. Any HDD enclosure is certainly larger.

It’s not the size of the NVMe enclosure that makes it clunky. It’s that you now have an extra dongle hanging off the back of your Mac and cluttering up the desk.

There is a reason for the popularity of those enclosure/hub combos that have the same footprint and color as the Mini.

Until someone brings out little two or four drive NVMe enclosures that fits exactly under the Mac Mini with a Thunderbolt bridge/plug that doesn't snag cables, because we all know Apple can't resist gauging buyers by refusing to include two easy to access M.2 bays on the underside.

I can't imagine anyone but Apple shareholders drooling at the taught of overpriced soldered memory would prefer a smaller Mac Mini case if ~0.5" more height would get you M.2 bays for storage.

these exist on amazon

You mean like this [1]? I would shocked if OWC didn’t have a version of this for the new mini form factor in the works.

[1] https://eshop.macsales.com/shop/external-drives/owc-ministac...

It even looks like the standard iconography for "Database"!

With heat now venting downward, that would work better above the new Mac Mini.

You might be interested in this then - https://satechi.net/collections/ssd-enclosures

Upgrade your memory and connect it externally over USB-C. It works brilliantly

Yes I have a Samsung T7 works like a charm.

Couldn't we just add extra drives into the extra internal SSD slots? Or does Mac Mini not have those?

There are no slots, it's all soldered directly to the motherboard. Even in the Mac Studio, which does use modular SSDs, they're proprietary modules rather than anything you can easily swap out yourself.

AFAIK, Apple took away those slots when Mac minis transitioned to Apple silicon. Attempts to replace SSD with re-soldering have not been successful.

So not even milling off the ssd works now?

Fortunately I don't really see the point of using a mac mini, so this doesn't bother me too much, but... it's poor taste. You're holding it wrong was not cool the first time.

You can mill them off and replace them with supported nands. People have videos on YouTube but it’s not very accessible to do.

The issue is that Apple moved the storage controllers into their SOC. So they use raw nand chips, and you need to use ones that the SOC supports.

You do have the option of a 10 gigabit Ethernet port, so you can build out a linux box for local shared storage with components as cheap as you're willing to trust.

That’s useful. The TB3 external 10 gig interfaces I’ve been using for my Mac get crazy hot.

As someone who just made a 16tb SSD array over Thunderbolt 3 (Best I could find) at 40gbps and the interface is still the bottleneck (disks are fast now!), 10gbps is going to feel really really slow vs the internal stuff.

> ... 10gbps is going to feel really really slow vs the internal stuff

How do you love your internet speed compared to the internal stuff?

It's possible to build a faster non-shared array if you aren't price sensitive (Thunderbolt 5 is 80 gigabits a secind), but someone with multiple computers and devices gets much better bang for the buck from shared local network storage.

As a bonus, you can back up your computers and iDevices to the shared local storage instead of paying for (probably much slower to access) cloud storage.

Idk idt over a GB/s transfer speed is bad, like 15m for 1TB? How often are we chucking a TB back and forth? Almost never.

Hell if I'm dumping cards from my camera or moving models across the network that's like 64-128GB max.

This is good news for me because I usually buy the base machine and accept its performance as a constraint on what I'm doing. I'm not sure it is all about AI though, Apple has been getting a lot of criticism for selling machines with just 8GB of RAM.

The RAM is expandable as well… however I am curious how well the extra RAM performs. Part of the M-series performance gain is from having the RAM dies very close to the processor.

It performs the same. You don't get any extra ram channels for upgrading the capacity.

I have a feeling this could simply be an outcome of samsung not offering anything smaller.

The M4 iPad Pro still starts at 8GB though, so Samsung is supplying them with lower capacity modules.

I learned yesterday that all M-chip Macs with enough RAM are getting Apple Intelligence?

This basically proves that Apple shot themselves in the foot for AI on mobile by artificially restricting RAM for so long! Heck, even the Neural Engine has turned out to be basically useless despite all their grandstanding.

So alas, their prior greed has resulted in their most popular consumer iDevices being the least AI compatible devices in their lineup. They could’ve leapfrogged every other manufacturer with the largest AI compatible device userbase.

I think it's great that Apple was able to ship devices that millions of people made happy use of without needing to put additional hardware resources into them. That's efficiency, not greed.

I own almost every Apple ecosystem device, but I definitely wouldn’t call their mobile device RAM capacity as sufficient. It physically hurts me when my iPad Pro M2 and iPhone 16 Pro Max (earlier 15,14,13,12,11) start to swap out live apps - sure some apps retain state, but the majority still don’t. Even Safari randomly reloads tabs for me, while I’m just researching purchases across <10 live tabs.

That's fair, it sounds like you've got a real problem. I'd be surprised of the majority of users experience that problem though.

No, they do. They just don't understand enough about technology/computing to know what's going on or they don't care (or at least pretend to, because they don't have any other experience).

It's really crazy that some android phones at half the price give a better browsing experience than many iPhones, especially the non "Pro" ones.

Yeah on my S24U rn I have 8 or so apps open, 100 or so tabs in ff (enough that it just shows an infinity symbol rather than number of tabs). That's with 12GB memory.

Tbf it seems to be browsers that kill memory on any platform. The Web is now just mobile breakpoints that load needlessly high res images.

Well that's not a cheap phone but yes if you are going high-end, it's probably a better experience than a comparatively priced iPhone.

Of course, the browsers are problematic for memory but that's not new and hilariously the first iPhone was supposed to work only with web apps. It's not like if they couldn't spend a few dollars more on their expensive premium hardware to guarantee a good experience for all use cases.

This is the big problem with Apple today, the milking at every single step, the stupid pricing ladder and scrooge attitude is extremely distasteful and completely unreasonable for the price asked.

> This basically proves that Apple shot themselves in the foot for AI on mobile by artificially restricting RAM for so long!

What they shot was us. My 14 Pro won’t do AI despite having a better NPU than an M1, all because Apple chose - intentionally - to ship it with too little RAM. They knew AI was coming and they did this anyway.

Although having played with it on my MBP it’s clear I’m not missing much. But still.

They knew they were releasing Apple Intelligence before ChatGPT went live? lol

They knew about LLMs in 2022. They've been shipping NPUs since transformers hit the scene in 2017.

Well given they're leaning on concepts like loras etc so much with apple "intelligence" I don't imagine they've been planning this for as long as you think they have.

And their npus weren't added in anticipation of LLMs, imo. You give em too much credit.

But now they get to sell new devices to them

This might actually push people to upgrade their hardware. And Apple retention rates are high. Apple will be fine.

Sounds like they've just done planned obsolescence faster to their lower paying customers.

It's not just RAM.

It's Unified RAM. So that memory is also used for the GPU & Neural Cores (which is for Apple Intelligence).

This is actually why companies moved away from the unified memory arch decades ago.

It'll be interesting to see as AI continues to advance, if Apple is forced to depart from their unified memory architecture due to growing GPU memory needs.

At this point it feels like (correct me if I'm am wrong) that Apple's AI is often performed "in the cloud". I suspect though that if Apple moves increasingly to on-device AI (as I suspect they will — if not for bandwidth and backend resource reasons then for privacy ones) Apple's Silicon will have adopted more and more specialized AI components — perhaps diminishing the need for use of off-board memory.

Last I checked, Apple was pretty much the only major player who does everything that they can do on device on device, that is their whole ethos behind it, no?

Yeah they have literature about this, they do as much as they can on device

It is possible that they do everything they can on the decide, but still have to do lots in the cloud, right? For some definition of lots, at least…

I mean, there is no need to speculate about any of this, they've put out a number of articles that outline their whole approach. I'm not really sure where the ambiguity lies?

They have a new bug bountry program for their confidential compute platform... in the cloud.

It's always local 1st and remote for certain things, and I think it warns your before going to the cloud IIRC.

>This is actually why companies moved away from the unified memory arch decades ago.

I don't understand - wouldn't the OS be able to do a better job of dynamically allocating memory between say GPU and CPU in real time based on instantaneous need as opposed to the buyer doing it one time while purchasing their machine? Apparently not, but I'm not sure what I'm missing.

I disagree that unified memory is a bad thing.

The usual reasoning that people give for it being bad is: you share memory bandwidth between CPU and GPU, and many things are starved for memory access.

Apple’s approach is to stack the memory dies on top of the processor dies and connect them with a stupid-wide bus so that everything has enough bandwidth.

If it's the shift I think you're referring to, I find it strange that you compare computing decisions from the 50s and 60s to today. You're correct, but that was over half a century ago. The reasons for those decisions, such as bus speeds, high latency, and low bandwidth, no longer apply.

Today, the industry is moving toward unified memory. This trend includes not only Apple but also Intel, AMD with their APUs, and Qualcomm. Pretty much everyone.

To me, the benefits are clear:

- Reduced copying of large amounts of data between memory pools.

- Improved memory usage.

- Generally lower power consumption.

Depart? They just got there, didn't they? And on purpose. There's more memory bandwidth, and also no need to copy from main memory to VRAM. Why would they bail on it?

I think they moved away because system memory was lagging behind in speed to the memory being used on video cards?

And besides, what Apple is doing is placing the RAM really close to the SoC, I think they are on the same package even, that was not the case on the PC either AFAIK?

Apple has an arm license but they still buy memory from Samsung (and others), it's not on the m chip die but it's provided by Samsung and then packaged above the m die.

Reply on news.ycombinator.com