Inference in this context means getting the output of a neural network on some input. Training meaning adjusting a neural network to so that it's outputs are more like the training data.

Training involves finding derivatives and requires higher precision calculations which is why they don't train using TPUs.