Hacker News Reader

Comment by simonw | Hacker News Reader

If you haven't tried it yet, and you have a modern enough phone, I strongly recommend trying out Mistral 7B on your mobile device.

I've been running it on an iPhone 15 using this app: https://llm.mlc.ai/#ios - App Store link: https://apps.apple.com/us/app/mlc-chat/id6448482937

The same team have an Android version too which I've not yet tried myself: https://llm.mlc.ai/#android

On my iPhone it works, and provides very decent performance. The only catch is that it needs pretty much the entire phone's memory to work - so if you switch out of it to another app and back again it resets the state and has to load the model from scratch.

View on HN

Apple has released a paper about running LLMs efficiently with low memory

https://arxiv.org/pdf/2312.11514.pdf

FYI that app is 3.3 GB on iOS, not your typical app size

I'd love to know why it's that big. I'm pretty sure it doesn't include the actual LLM model files - the first time you use a model you have to click to download it, and those are likely even larger than 3.3GB.

It comes with Mistral 7B, Llama 7B and RedPajama are available for download.

I've been playing with it on the recent iPhone Pro, and it is pretty fast. The mistral 7B is very wordy though and is prone to making up stuff. It's fun to talk to, but it's not very useful currently.

Based on this demonstration of a local LLM, with optimizations, it definitely the way of the future on apple devices at least.

> so if you switch out of it to another app and back again it resets the state and has to load the model from scratch.

This has been my experience anyway ever since upgrading to iOS 16 (and now 17) anyway. Everything is always paging, even on the latest hardware.

There is no paging in iOS. At least paging in the Linux sense. Your apps memory will get evicted when you switch to another app if the system needs it.

It is up to each app to store their state and resume. One of the most annoying apps for not doing this well is Google Maps. Holy crap I hate when I search something and get a route set and tab out to check something and com eback to a fresh Google Maps like 5 seconds later. They will fix it for a while then break it again several times a year. Once you start navigating it seems pretty reliable but until you hit start it is a shitshow.

The Google Maps app is atrocious on iOS. So is their website.

Other than not always storing state the Google Maps app works quite nicely on iOS.

I just switched to Apple Maps and have been pretty happy with it. Uninstalled Google Maps. Telling me that in 2010 would make me laugh at you.

Apple Maps is terrible at discovery which is what I am doing with Google Maps 90% of the time.

I really want to like it, but it's pretty terrible at many things imho.

I've tried really hard to switch to Apple Maps but even after three months it simply can't seem to remember I basically drive to 5-6 addresses on a weekly basis. It's always to the parking lot near the office on Monday and Wednesday morning, daycare to pick up the youngest on the way back, swimming pool on Saturday, etc. etc. but navigating to those addresses is always a pain (even selecting it from recent addresses in CarPlay) and it did sent me to the same street as the daycare in a completely different town several times now.

While I value not giving more data to google, after three months I decided it simply wasn't worth it. I found the experience (especially the first time it sent me to a completely different town I never go to and I didn't catch it) pretty infuriating.

I'd recommend setting up some Shortcuts with the addresses that can be triggered from Siri. For example, "Siri, we're going to the office", "Siri, let's go swimming",

One nit with this is that I haven't been able to make it automatically transition into navigation.

do you happen to have a link where I can read more about this?

Yes, I’m aware. I shouldn’t have used that specific verbiage but that was what I meant. Which is worse than paging from a UX perspective since in practice no app does it perfectly (esp if there’s unsaved user input in a textbox).

Mistral is amazing for its size.

Reply on news.ycombinator.com