Show HN: OfflineLLM: Live Voice Chat with DeepSeek, Llama on iOS and VisionOS

Hi, this is something I've been working on for the past 18 months. There are an abundance of tools to run LLMs locally on desktops (e.g. ollama, LM Studio), but other devices have been left out. This is has been a project to run these models onto iOS and visionOS, which has turned out to work really well. Even an iPhone 14 Pro can quite easily run the 3B parameter version of Llama 3.2. CLIP models also work well too!

It also has a Live Voice Chat which gives a 2-way conversation experience, functionality similar to the cloud-based Gemini Live feature that Google offers.

Under the hood it can run most GGUF models, using a heavily forked and diverged verison of llama.cpp which has helped performance on the mobile devices.

The next steps are to integrate Apple's on device 3B model which hopefully they will be opening up access to at WWDC in a week's time. I'm also in the middle of adding in support for Gemma 3 and Qwen 3.

Let me know what you think!