Homomorphic encryption doesn't exit at any meaningful scale. The weights must still be stored plain in the RAM for inference to work