The neural network inference (and training) is all done in compiled C libraries called by Python. The inefficiencies you're seeing are because you're doing it on a CPU, probably in one thread, rather than on a GPU with tens of thousands of "threads."