GPT-J is trained on TPU VM generously offered by Google as part of their TPU research cloud. It’s a small version true but the results are almost on par with reported GPT-3

Source - https://github.com/kingoflolz/mesh-transformer-jax