Hacker News Reader

Comment by p1esk | Hacker News Reader

Are you using FP16?

No. We weren't sure if that'd be a good idea since it wasn't trained with low-precision, and 345M thankfully didn't require going that far. 744M might, though. (Another option is model parallelism since I have 2 GPUs and that might be enough, perhaps freezing more layers and training incrementally, or reducing the 1024 token window to smaller ones like 700.)

On the NVIDIA GPT- 2 implementation:

>What would be the largest model one could train across 2x 2080Ti?

>~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations

https://twitter.com/TheRealRPuri/status/1161322580126470145

They haven't released such models, though, and I don't know if it would be drop-in compatible with the OA GPT-2-774M checkpoint (they're training their own GPT-2s using their own webtext corpus).

I haven't look into at all myself, but he also said:

>We do provide training code that should work out of the box for gpt2 117M/345M

https://twitter.com/TheRealRPuri/status/1161319745259393024

It would take forever (or $$$) to train even 117M model from scratch.

I read that meaning you can start with the actual pre-trained GPT-2 models but I never got an answer when I specifically asked if that was the case.

You maybe should try tensorflow automatic mixed precision! https://github.com/zihangdai/xlnet/pull/200

fp16 saves a lot of memory and is worth doing. I've not had trouble fine tuning all these models with fp16.

Have you fine tuned 774 successfully using a single GPU?

I recommend Nvidia Apex, it offers several ways to mix precision.

Reply on news.ycombinator.com