I took a look at Nvidia Triton Inference server, and it might be a good option for production especially as it has c++ api.

Amazing feedback, Thanks!