Have a look here: https://github.com/IntelAI/OpenVINO-model-server/blob/master... You can replace tf-serving with OpenVINO to get even better performance and latency when running on CPU
Even basic image classifiers tend to be 100x faster on a GPU or TPU...