Hacker News Reader

Comment by itake | Hacker News Reader

I am cautiously optimistic, but the reality is these models are huge. LLama 7b is 13g. Even if there is enough cpu cycles, the phone's storage can't support multiple models (and there will need to be multiple models).

Compare that to cloud, you can run the latest and greatest model and effectively unlimited storage.

View on HN

But the cloud would cost apple money, while if ran in your own device they can sell you the hardware for a healthy margin, and market privacy. I think a single useful model on device or many tiny ones are feasible. Even today, the photos app on my iphone knows what pictures are dog pictures, can remove backgrounds, etc.

It would simply become a part of the iCloud+ bundle and contribute to service revenue.

Hell, that’s likely what will happen anyway. The models run on device with cloud assistance.

Inference/RAM is 100% the bottleneck, not storage, and it isn't close.

These models all have to be fully loaded in memory to run. I think the max RAM on an iPhone is 8GB, max storage is 1TB, so we're talking 2 orders of magnitude more ability to store models vs actually running them.

And of course you'd really rather not use the full RAM for inference.

You're 100% correct on the technical challenges here, but the fully loaded requirement is what Apple is trying to address as per the paper they just published. I'm somewhat skeptical that they've solved it since it's a hard problem. We'll see!

I'm currently running dolphin 2.2.1 mistral on an M1 Mac Mini with 16GB of RAM. Zero optimisations, just running base LM Studio, I haven't even bothered with GPU acceleration.

It thinks about 5-10 seconds before answering and writes the answer about the speed I can read it.

The current top of the line iPads are more powerful than this and an optimised modern iPhone implementation using the ML hardware should be faster than my current setup.

A reasonable quantization (Q4_K_M) is 4GB on disk and uses 6.5 GB of RAM. It will run at reasonable speed on an iPhone 15 Pro today. Using a smaller model or a more efficient quantization on a phone two years from now (aka whatever's in their lab today) will be easier than running the latest First Person Shooter.

Welcome to the future.

Edit: I see a few comments about hundreds of GB. That works for storage, but DRAM consumes battery just by existing, and that's been keeping RAM sizes from increasing the way storage has.

Just as an example, my phone has 512gb of onboard storage and 16gb of RAM. I think it is safe to say that in the not-to-distant future, we will see the option to run multiple models on device with only a slight storage hit.

13 GB a lot? What year is it? 1995? If Apple or Google wanted to add 1 TB to their phones, it wouldn’t cost them anywhere near what they charge us to add 256 GB.

A copy of GTA V takes like 100 GB.

Reply on news.ycombinator.com