If you're loading the model for every prediction, you're doing something really wrong. Pre-processing and post-processing are indeed usually trumping the core computation itself, but as you mentioned it's not done in the NN.
Could be something that is just one once per process or the like. But yeah, if you are doing multiple inferences it would be silly to load the model each time.