Tinygrad is python only right? Can it provide gradients during C++ runtime as well? ONNX runtime have multiple language backends for inference.
One option to your case is OpenVino. It's written in C++ and has Python Bindings. Also, it can be used to train new nets. You can use ONNX files with OpenVino too.