Hacker News Reader

Comment by mark_l_watson | Hacker News Reader

A few months ago I bought a Mac with a M2 Pro 32G, and it is great for LLMs. I never thought that I would be running models like mixtral-8-7B-q3 locally on my own computer. I think the M2 Pro has 200GB/sec internal bandwidth, about 25% faster than the M3.

I think smaller models will keep getting better. For privacy and economic reasons, it makes sense to do as much AI on-device as possible.

View on HN

But is it stable? I've seen a video on YouTube [0] where someone was showing LLM inferencing on a MacBook compared to an NVIDIA card, and it was really good, but before the video was over, everything on the Mac stopped working (he was also recording via OBS I think and it went to a halt).

This was a MacBook, but since you're just saying "Mac", I don't know what you're referring to.

I think that the MacBook couldn't cope with the heat generated so it went to its knees. I've read about it on Reddit as well.

[0] https://youtu.be/jaM02mb6JFM?t=411

I bought a Mac Mini specifically for experimenting with LLMs. On the rare times I want to try fine tuning, I use Colab Pro or rent a server on Lambda Labs. I just use the M2 for inferencing, so far. So, no heat problems.

EDIT: maybe the YouTube demo was using 2 bit quantization?

I've heard multiple reports that you can hard-lock a M1/2 Mac simply by using OpenGL in harsh ways, so I wouldn't be surprised that an LLM + OBS can crash it.

I suspect OBS can crash it all by itself.

I set my kids up with Faraday.dev on their M2 Airs and it does just fine. They will have chatbots up for hours to help them with schoolwork or when playing Minecraft to give them ideas.

The big thing is just don’t lock the models into memory on lower RAM systems. That gets you into trouble when some of the unified RAM is needed for something else and you can soft lock the system. The heat is not an issue though in my experience.

I use my MBP for local ML and then CUDA in production. So far so good. We run YOLO inference models and Open CLIP models. Soon adding SD1.5 to the mix!

Can you recommend any primers about running small models on-device, training/fine tuning on particular datasets (e.g. a specific code base), etc?

https://github.com/ml-explore/mlx

https://github.com/ml-explore/mlx-examples

This is where all of the action is at right now.

You can also use LM Studio: https://lmstudio.ai/

An open source lmstudio software released recently, but I'm having trouble finding a link.

you might be thinking of GPT4ALL?

https://gpt4all.io/index.html

I struggled to get it working on my machine, it kept corrupting the models as it downloaded them.

Was it this? I haven’t tried it yet but it does look nice.

https://jan.ai/

Yes! That was it.

I maintain this one for native macOS chat https://github.com/psugihara/FreeChat

I tried everything for running small models locally, and finally settled on ollama.ai - good command line and REST API functionality. If you like Lisp, recent updates to my Racket and Common Lisp books have examples using the Ollama REST APIs (Can be read online).

I've had a similar experience with lm studio and draw things on my Macbook.

Reply on news.ycombinator.com