Hacker News Reader

Comment by tkems | Hacker News Reader

I think that running AI on device is the clear next step in the world of super powerful portable phones/devices. I have a 2021 model phone with very powerful specs (granted it is the high end model that I got used). With nearly the same RAM and storage as some of my laptops from a few years ago, I see that Apple might want to offload some of their expensive AI apps to these powerful user devices.

I see this also as a privacy win as I don't want any of my data being used for training. I also like to have control over what models I use and the ethics the models follow.

View on HN

I am cautiously optimistic, but the reality is these models are huge. LLama 7b is 13g. Even if there is enough cpu cycles, the phone's storage can't support multiple models (and there will need to be multiple models).

Compare that to cloud, you can run the latest and greatest model and effectively unlimited storage.

But the cloud would cost apple money, while if ran in your own device they can sell you the hardware for a healthy margin, and market privacy. I think a single useful model on device or many tiny ones are feasible. Even today, the photos app on my iphone knows what pictures are dog pictures, can remove backgrounds, etc.

It would simply become a part of the iCloud+ bundle and contribute to service revenue.

Hell, that’s likely what will happen anyway. The models run on device with cloud assistance.

Inference/RAM is 100% the bottleneck, not storage, and it isn't close.

These models all have to be fully loaded in memory to run. I think the max RAM on an iPhone is 8GB, max storage is 1TB, so we're talking 2 orders of magnitude more ability to store models vs actually running them.

And of course you'd really rather not use the full RAM for inference.

You're 100% correct on the technical challenges here, but the fully loaded requirement is what Apple is trying to address as per the paper they just published. I'm somewhat skeptical that they've solved it since it's a hard problem. We'll see!

I'm currently running dolphin 2.2.1 mistral on an M1 Mac Mini with 16GB of RAM. Zero optimisations, just running base LM Studio, I haven't even bothered with GPU acceleration.

It thinks about 5-10 seconds before answering and writes the answer about the speed I can read it.

The current top of the line iPads are more powerful than this and an optimised modern iPhone implementation using the ML hardware should be faster than my current setup.

A reasonable quantization (Q4_K_M) is 4GB on disk and uses 6.5 GB of RAM. It will run at reasonable speed on an iPhone 15 Pro today. Using a smaller model or a more efficient quantization on a phone two years from now (aka whatever's in their lab today) will be easier than running the latest First Person Shooter.

Welcome to the future.

Edit: I see a few comments about hundreds of GB. That works for storage, but DRAM consumes battery just by existing, and that's been keeping RAM sizes from increasing the way storage has.

Just as an example, my phone has 512gb of onboard storage and 16gb of RAM. I think it is safe to say that in the not-to-distant future, we will see the option to run multiple models on device with only a slight storage hit.

13 GB a lot? What year is it? 1995? If Apple or Google wanted to add 1 TB to their phones, it wouldn’t cost them anywhere near what they charge us to add 256 GB.

A copy of GTA V takes like 100 GB.

I wager there is literally zero chance that Apple will let you have any freedom in the model used, and there's zero chance that you'll be using anything except the Siri_Model unless you hack your iDevice.

Just like there is 1 app store and 1 assistant, Apple will fight forcefully to make sure there is 1 AI model.

Right now there are already local LLM apps with uncensored models in the app store.

I think there are two things that you can’t do on iOS:

- have “Hey Siri” send audio to your code

- have your code reliably listen in to the microphone so that your code can listen for its own trigger phrase (AFAIK you can start an audio recording session, and have that continue while your app is in the background, but that requires users to open your app first. I also am not sure that recording will reliably survive device sleep and automatic app shutdown because of memory pressure)

I think #1 is fine. It doesn’t seem fair to me that Apple should be forced to allow other developers link “Hey Siri” to their voice assistant because that assistant may be detrimental to the value of the “Hey Siri” mark or Apple’s values, for example because it is racist, sexist, homophobic, etc.

The second IMO isn’t fine, but I can see Apple argue they are in their rights to have some control over how that rigger phrase gets detected. “Hey Siri” uses a small on-device network specifically trained for that, in order to conserve battery life (https://machinelearning.apple.com/research/hey-siri), and they may argue having good battery life is essential to their brand.

Is Apple going to ship these LLMs as part of the OS? I expect app developers to bundle them in their apps. Also Llama has a specific license clause that it’s available to anyone except Apple and Google

you can already run llms on iphone locally https://llm.mlc.ai/

I find it very difficult to believe that Apple is going to incorporate AI hardware into their device and then open it up to everyone freely. That would be very off brand. Even in the browser space Apple famously forced all competing browsers to use Apple Webkit as their engine. They certainly have history around lockdown and lock-in.

The CoreML APIs let you make use of the neural engine that’s built in.

It doesn’t benefit Apple to restrict access to silicon.

https://developer.apple.com/documentation/coreml

> I find it very difficult to believe that Apple is going to incorporate AI hardware into their device and then open it up to everyone freely.

The Core ML API for using the neural processing features of Apple's chips has been in there for a while and plenty of developers are using it.

They don’t have to open it up freely. They could choose to add an API with a chat completion type of interface, whisper-like, etc.

Nothing would need to change in the existing terms and conditions with developers: we only accept binaries that use our official APIs.

EU is already demanding that Apple allows more app stores, isn't it?

Well, then if you can convince the EU to force Apple to allow uncensored models that will happily teach you how to make bombs and all manner of things like that, then you can have those models on your iDevice. Not sure the EU is going to jump on board that kind of free speech though.

They already allow uncensored web browsers that will happily open up web pages about making bombs.

You could make the same (unconvincing) argument for apps on third-party stores, teaching the same dangerous things or doing even worse things like (gasp) allowing pornography :P

Generally speaking, Apple has maintained rigid control over apps and have banned everything you mentioned.

If EU app stores fill up with pornography and terrorism apps, you can imagine their move will be decried as a debacle and used by Apple to demonstrate how the EU failed by forcing them to open up

If EU app stores fill up with pornography and terrorism apps, that will be a problem of the companies running the stores, not Apple's problem.

Apple has no right to decide what users do with the hardware they sell.

Can we start expanding this argumentation to Microsoft, Sony and Nintendo too?

I really want to start playing hardcore porn games on my PS5 as soon as possible. Sony has no right to decide what I do with my hardware, right?

Can we start expanding this argumentation to Microsoft, Sony and Nintendo too?

Sure.

I really want to start playing hardcore porn games on my PS5 as soon as possible.

And if somebody figures out how to do that, they should be able to.

Sony has no right to decide what I do with my hardware, right?

Pretty much, yes.

We don't need to "figure it out". We need the EU to force the big three console manufacturers to allow 3rd party stores in their consoles.

We can do that after the big fishes, those devices aren't such a big deal which is why people aren't complaining as much about them.

> Can we start expanding this argumentation to Microsoft, Sony and Nintendo too?

Absolutely.

> Sony has no right to decide what I do with my hardware, right?

That is correct.

Can you allow uncensored models but they need to be wrapped around Apple's on-device censorship firewall?

Or is there a common API for llm constitution (or however it is called)?

Apple already has the CoreML framework anyone can use to do on device processing. They even have tools to help convert things into the format the iPhone uses. As others have pointed out you can already find apps with public models on the Apple App Store.

They’re not doing anything to censor any of this.

At least they'll let you say "duck" now.

My next step at least is various models running at home that i can talk to remotely with my phone. Join a discord channel and talk to my bots. My bots respond or do things back in the channel or with whatever else they have access to.

Reply on news.ycombinator.com