Hacker News Reader

Local AI sounds nice but most of Apple’s PCs and other devices don’t come with enough RAM for a decent price needed for good model performance and macOS itself is incredibly bloated.

View on HN

That's true for current LLMs, but Apple is playing the long game. First, they are masters of quantization optimization (their 3-4 bit models perform surprisingly well). Second, Unified Memory is a cheat code. Even 8GB on M1/M2 allows for things impossible on a discrete GPU with 8GB VRAM due to data transfer overhead. And for serious tasks, there's the Mac Studio with 192GB RAM, which is actually the cheapest way to run Llama-400B locally

The Mac Studio comes with up to 512GB of unified memory now.

Depends what you are actually doing. It's not enough to run a chatbot that can answer complex questions. But it's more than enough to index your data for easy searching, to prioritise notifications and hide spam ones, to create home automations from natural language, etc.

Apple has the ability and hardware to deeply integrate this stuff behind the scenes without buying in to the hype of a shiny glowing button that promises to do literally everything.

That might work well for Apple to be the consumer electronic manufacturer that people use to connect to OpenAI/Anthropic/Google for their powerful creative work.

Reply on news.ycombinator.com