On the NVIDIA GPT- 2 implementation:
>What would be the largest model one could train across 2x 2080Ti?
>~800M gpt2. this is largely due to the memory required to House parameters + optimizer states. If one uses a smaller optimizer than Adam training something larger should be possible. Make sure to turn on activation checkpointing with —checkpoint-activations