Oh that's unfortunate, can't the models be exported to pytorch through e.g onnx?
Even small models can be a headache to export if they have even they use anything custom. I can't even imagine something the size of GPT-3.
The GPT-2 models included with Transformers can export to ONNX fine w/ a helper included with the Python onnxruntime.
Larger models aren't really more complicated than smaller ones though. GPT-2 is already supported, I believe only difference with GPT-3 is sparse attention.
To run in PyTorch the model architecture must be ported.
ONNX is slightly different; you could export the model to run in the onnxruntime but that has tradeoffs.