Hacker News Reader

Comment by Valgrim | Hacker News Reader

Is this the hardware used to build the model, or to run the model? Does this single 100000$ computer run the entire chatGPT service concurrently for all users around the world? If so this seems incredibly cheap.

View on HN

Supercomputer used for training, top 5 in the world apparently:

https://news.microsoft.com/source/features/ai/openai-azure-s...

I would guess that's the minimum needed to run an 'instance' of the model for inference

not sure how many concurrent users that could support, but I'd imagine the public service has a whole bunch of these racks

from what I read... GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB)

and then GPT-3 is 175B params or ~29x larger, so maybe needing 7-800GB RAM

8x 80GB A100s would give 640GB RAM, so its approx the right ballpark

> GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB)

You can load that on M1/M2 Macs, which have unified GPU memory up to 128GB.

The magnificent part is that the 8xA100 is only a blade and you can fill a whole rack with them. Yet one is already incredibly powerful.

Also the successor H100 is already released and at least 2x as powerful.

Also the successor H100 is already released and at least 2x as powerful.

Checks eBay for used A100s...will not be getting one soon. :-)

In machine learning, generally the expensive part is training the model, not running the model.

At this point the costs of hosting the trained model probably surpass the training cost.

has anyone numbers on how many queries/second in inference time chatGPT can serve?

It might, I'm not sure. It depends on many factors: how much cloud discount are they getting, for example? And do you count the cost of training all previous models before chatGPT, or just the cost of training chatGPT itself?

But you might be right, chatGPT is insanely popular right now.

Reply on news.ycombinator.com