Hacker News Reader

And in 2029 people would call 10 trillion params models "medium size"...

I mean, with about $120K, we can technically run a 137B model locally. Any small company or rich individuals can afford this easily. So the "outrageous" back then is pretty "middle" now.

View on HN

I can run a 30B parameter model on my M1 laptop at a pretty decent token rate. I can run two instances of the same model in parallel without affecting the speed.

A maxed out M2 Mac Studio can easily run 70B models and very likely 137B models too. The key is quantization.

So it’s all within reach for under $7k if you’re willing to accept a slightly lower quality (~5 bit quantization).

Reply on news.ycombinator.com