Hacker News Reader

Hacker News

Comment by reedlaw | Hacker News Reader

Parent Full thread

reedlaw

In case you're wondering what it takes to run it, the answer is 8x H200 141GB [1] which costs $250k [2].

1. https://github.com/MiniMax-AI/MiniMax-M1/issues/2#issuecomme...

2. https://www.ebay.com/itm/335830302628

View on HN

That's full quantization. If you run Q4 or Q8 you can run this on <$10,000 equipment.

My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.

>My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.

People have tested it. Q8 has essentially no drop in quality, Q4 is measurable but still not realistically a problem. If this impacts you, just pay for the commercial saas option.

This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs.

>This assumes that the benchmarks are representative of real usage scenarios. I'm not saying that there is bad faith, but that benchmarking is really hard in the context of LLMs.

It's a fair point, but the conclusion is 'i dont know'

I could assume that it gets better because it'll keep to simpler code.

No point in running anything but full quantization.

Quantization doesn't work? Really?

And if you add in heavy sparsification it should fit and run on a raspberry pi.

So in around 6 months, we will see that the person who bought this H200 in the listing just got scammed for $250k and will realize that you just needed specific quantizations to the model and a few optimizations to run locally.

Unless they want to train their own model, buying this for inference for $250k is unnecessary and still isn't enough for a full production deployment.

It's already sparsified from the 150T parameter model..

It took me several hours to realize that 150 trillion parameters is a reference to the number of synapses in a human brain.

Can’t you run it on a Mac Studio with 512GB? That’s about $8,500.

It's also 1/20 of the speed. So not very useable.

How many parameters does this model have?

456 bn, about 46 bn active at a time (it's moe)

Reply on news.ycombinator.com