GPT-J for text generation: hardware requirements | Hacker News Reader

GPT-J for text generation: hardware requirements

10 pointsjuliensalinas5 comments

Since the release of GPT-J, I worked hard in order to add it to NLPCloud.io for text generation.

This is done now and the infrastructure is stabilized but that was tricky. So I thought I would share here my key takeaways, in case it can help some of you:

- On CPU, the model needs around 40GB of memory to load, and then around 20GB during runtime.

- On CPU, a standard text generation (around 50 words) takes approximately 12 CPUs for 11 seconds

- On a GPU, the model needs around 40GB of memory to load, and then around 3GB during runtime + 24GB of GPU memory. For a standard text generation (around 50 words), the latency is around 1.5 secs

The 2 main challenges are the high amount of RAM needed for startup, and then high amount of GPU memory needed during runtime which is quite impractical as most affordable NVIDIA GPUs dedicated to inference, like Tesla T4, only have 16GB of memory...

It's very interesting to note that, during my tests, the latency was pretty much the same as GPT-Neo 2.7B on the same hardware, but accuracy seems of course much better.

If some of you also ran these kinds of benchmarks on GPT-J I'd love to see if we're aligned or not!

This is useful information, but it would even more useful if it was mention which CPUs were used - knowing how long something took with 12 CPUs without knowing their model, cannot be used to guess how long it would take with any other CPU(s). It is also not clear if cores were confused with CPUs - 12 CPUs with many cores each or a single 12-core CPU is a very big difference.

By the way, according to this it is possible to minimize VRAM and RAM usage significantly: https://github.com/arrmansa/Basic-UI-for-GPT-J-6B-with-low-v... ("a repository to run GPT-J-6B on low vram systems by using both ram, vram and pinned memory"):

I did not tried this myself yet, but here are their results on two different systems:

System 1: 16 gb ddr4 ram . 1070 8gb gpu. 23 blocks on ram (ram_blocks = 23) out of which 18 are on shared/pinned memory (max_shared_ram_blocks = 18). timing - single run of the model(inputs) takes 6.5 seconds. 35 seconds to generate 25 tokens at 2000 context. (1.4 seconds/token)

System 2: 16 gb ddr4 ram . 1060 6gb gpu. 26 blocks on ram (ram_blocks = 26) out of which 18 are on shared/pinned memory (max_shared_ram_blocks = 18). timing - 40 seconds to generate 25 tokens at 2000 context. (1.6 seconds/token)

Thank you for this data. I'm no machine learning engineer, but learning some NLP and was wondering what kind of computer would get to run state-at-the-art models.

I assume the hardware requirements for training are massive and impractical for home-use, isn't it?

Yep totally. You can forget about training Tranformer-based NLP models from scratch at home, sadly... however you can choose a pre-trained model and fine-tune it with your annotated data.

Serving the models for inference is different, you can definitely perform inference on your local machine for most of the existing NLP models, even Transformer-based models. But GPT-J and GPT-Neo might be exceptions though as they require tons of RAM.

Also, even if you can perform inference at home, you might not have great perfs without NVIDIA GPUs dedicated to AI.

Ah here I was - a total amateur eager to learn more - hoping to add a TPU unit on a rpi cluster to run it.

(thanks for sharing!)

Lol, you're welcome ;)

Reply on news.ycombinator.com