What is the benefit of having all of the ML bits on device? Can models leverage them post training?
Yes, the whole point is to be able do to things like improve personalized speech recognition on-device, image recognition on-device, translation on-device, etc.
Potentially improved privacy (this is how Apple tries to sell it). Less data has to leave the phone to gain the utility of the ML models.
Improved device performance. Reduced network use and better specialized chips leading to better performance. Either in terms of better battery life or better time to get the result.
On a Google device I would say the potential is reduced to zero!
Yes. The whole point is that you have a slow process train the model offline on very large volumes of sample data, then use that trained model to make actual inferences based on data you find in the field. As those models become bigger and more complex, it takes progressively more computing power to run inference on those models. These ML accelerators are effectively the new GPUs — highly specialised processors designed to more efficiently handle highly specialised workloads.