How fast would ChatGPT run on a laptop locally?

3 points·owenpalmer··9 comments

Obviously it depends on the laptop, but that's not the point. Would it be extremely fast? Extremely slow?

It wouldn't[0], at all[1]

[0] https://bdtechtalks.com/2020/09/21/gpt-3-economy-business-mo...

[1] https://lifearchitect.ai/chatgpt/#local

It's quite unlikely that you would even be able to run it at all. The model is quite large and probably won't fit in memory. If you could get it to run, it would be extremely slow

Worth checking out: https://github.com/karpathy/nanoGPT The associated videos will go into this more, iirc, at the end he said the real GPT-2 would take 5-20 seconds to generate a small amount of text using CPU only.

Just fine. GPT-Neo-125m can generate ~200 tokens in 10 seconds. Larger models like 1.3b use more memory (~6gb) and take longer (~20-30 seconds), but produces somewhat more coherant answers.

Seriously though, the answer is "better than you think". Go boot up PyTorch and see for yourself!

ChatGPT is built on GPT-3 which is 175 billion parameters or roughly 300GiB VRAM.

Significant difference between smaller models.

Yep, I'm merely pointing out that pruned GPT models do exist, and work on laptops. The technology does scale.

ChatGPT is GPT-3++, the one people are thinking of is 175B parameters, post-trained on a very curated dataset to get the chat like behavior

At the local speeds, my time is worth more than what OpenAI charges

That's fine, but it's also a far-cry from "GPT doesn't run on your laptop". They're asking about how it scales, I'm giving them an answer.

Yes, you can also use API requests to get the same results, but that's not running locally.

OP asked about ChatGPT, which is not GPT-2, it is GPT-3.5 based with a secondary training regime

the question depends on how much data are you going to use it with

Reply on news.ycombinator.com