Hacker News Reader

Comment by TheAceOfHearts | Hacker News Reader

Does anyone know why they haven't released any 30B-ish param models? I was expecting that to happen with this release and have been disappointed once more. They also skipped doing a 30B-ish param model for llama2 despite claiming to have trained one.

View on HN

I suspect 30B models are in a weird spot, too big for widespread home use, too small for cutting edge performance.

For home users 7B models (which can fit on an 8GB GPU) and 13B models (which can fit on a 16GB GPU) are in far more demand. If you're a researcher, you want a 70B model to get the best performance, and so your benchmarks are comparable to everyone else.

I thought home use is whatever fits in 24GB (a single 3090 GPU, which is pretty affordable), not 8 or 16. 30B models fit.

While some home users do indeed have 24GB of vram, the fact is a 4090 costs $1700

Such models will never top the number of downloads charts, or the community hype, as there’s just loads more people who can use the smaller models.

And if you can afford one 4090 you can probably afford two.

Why 4090, though? I read (and agree) that 3090 is generally considered to be the best bang for the buck: 24GB, priced at $800-1000 range, and giving decent TPS for LLMs.

Maybe they think more people will just use quantized versions of 70B.

Why should they?

Unless I'm misremembering, they announced it at one point. It's just giving people more options.

That was for version 2, not 3 or 3.1, if I recall correctly.

Reply on news.ycombinator.com