Hacker News Reader

Do you have a ballpark idea of how much RAM would be necessary to run llama 3.1 8b and 70b on 8-quant?

Roughly, at Q8 the model sizes translate to GB, so ~3 and ~70GB.

You mean 8, not 3?

Yes, apologies, can't edit now