Hacker News Reader

Anyone on HN running models on their own local machines, like smaller Llama models or such? Or something else?

Doesn’t everyone? X) it’s super easy now with ollama + openwebui or an all in 1 like mlstudio

Was just concerned I don't have enough RAM. I have 16GB (M2 Pro). Got amazing mem bandwidth though (800GB/s)

M2 Pro has 200GB/s

You're right.

M1 and M2 Pro: 200GB/s

M3 Max: 300GB/s

M1/M2 Max: 400GB/s

M1/M2 Ultra: 800GB/s

Seems to be the case. An Ultra .. wow. But 200GB is also still good, so not complaining.

It's on the low side but plenty for something like this new 3b model. Anything up to 8GB and you've still got as much as a base model Air left over.

For sure dude! Top comment thread is all about using ollama and other ways to get that done.