Hacker News Reader

Has anyone tried this? I have 96GB of GPU memory; will that be enough to run the biggest model?

I have not tried, but 96GB of GPU memory is plenty, for inference there should certainly be no issue. Their biggest model has 13B parameters, you should be able to run inference (float16) already with 32GB of memory.

With 96GB of memory you should also be able to fine-tune it (possibly some tricks like gradient accumulation and/or checkpointing might be needed), but you have to be ready for many days of computation...

> but you have to be ready for many days of computation...

I was thinking since we have API prices in tokens and now it looks like self hosted inference on high end GPUs for similar models. Then based on electricity prices there will be a self-hosted prices in tokens. Then how close are these already? What is the markup today from roughly the raw electricity cost that OpenAI has.

13B fits nicely even in a 3090 (24gb vram)!

Only if you can use int8, which may or may not be convenient.

with load_in_8bits ?

Does that mean I would be able to run it on an M1 Max with 64 gb unified memory?

Reply on news.ycombinator.com