Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by htrp | Hacker News Reader

Parent Full thread

htrp·

maybe we'll look back at token context windows like we look back at how much ram we have in a system.

View on HN

I agree with this in the sense that once you have enough, you stop caring about the metric.

And how much RAM do you need to run Mixtral 8*22B? Probably not enough on a personal laptop.

Generally about ~1gb ram per billion parameters. I've run a 30b model (vicuna) on my 32gb laptop (but it was slow).

I run it fine on my 64gb RAM beast.

At what quantization? 4-bit is 80GB. Less than 4-bit is rarely good enough at this point.

Is that normal ram of GPU ram?

64GB is not GPU RAM, but system RAM. Consumer GPUs have 24GB at most, those with good value/price have way less. Current generation workstation GPUs are unaffordable; used can be found on ebay for a reasonable price, but they are quite slow. DDR5 RAM might be a better investment.

While there is a lot more HBM (or UMA if you're a Mac system) you need to run these LLM models, my overarching point is that at this point most systems don't have RAM constraints for most of the software you need to run and as a result, RAM becomes less of a selling point except in very specialized instances like graphic design or 3D rendering work.

If we have cheap billion token context windows, 99% of your use cases aren't going to hit anywhere close to that limit and as a result, your models will "just run"

I still don’t have enough RAM though ?

RAM is simply too useful.

Reply on news.ycombinator.com