Hacker News Reader

Comment by onion2k | Hacker News Reader

Running models on-device on a Mac is immensely annoying though. Figuring out what will work out of BF16, FP8, BF16+FP8, NVFP4, INT8, GGUF ... the list goes on ... is 'non-obvious' at best. Apple do little to support with tooling. There's MLX, but unless you're happy to transform a model to that format yourself you'll be lagging a long way behind.

Apps like LMStudio, Ollama, Draw Things, etc do a great job of simplifying it but it's still a pain.

View on HN

How is it a pain exactly? It’s just learning and only takes a day or two to get up to speed. We seem to have forgotten that for the past fifty years doing all kinds of tasks on computers has been tedious and involved and time consuming to even get working. My first computer had 48kb of RAM and to play a game you had to load it off cassette for five minutes. That was annoying. Having LM Studio download a model and load so you can chat or attach an agent it is effortless and easy in comparison.

Not to mention you can just ask Codex or Claude Code to set up a local LLM for you. Use one tool to set up the other.

I dunno I use LMStudio pretty regularly and the MLX folks and the community usually have MLX versions of new model releases up within a day or two.

For some models like the popular coding and chat models, things move faster. For things like images, voice, sound etc they definitely lag a long way behind.

& they clearly delineate all the models that’ll work on your exact machine (but guardrails can be disabled in settings)

Converting a model to mlx is literally a git clone and mlx_lm.convert. 5 minute operation after clone.

Or just ask any model to do it for you.

Counterpoint, I'm just a happy user of this local LLM [1] from antirez (of Redis fame).

[1] https://github.com/antirez/ds4

What problems are you running into? Ive been running multiple qwen coder next agents locally and it works fine.

In general I just don't like not using a cross platform architecture of some kind.

MLX is fine, but the cross platform alternatives (ONNX) are terrible, GGUF often lacks and doesn't have the "easy convert" that some of the commenters below say MLX has.

Reply on news.ycombinator.com