I don't understand. Time-to-train is always a product of model and the amount of hardware you use. TPUs are fast and cheap allowing you to use more hardware for the same cost. The TPU design also makes it scale quite well.

If you really wanted to get from 48 hours to multiple experiments per work day, you could (probably) do that right now by scaling horizontally. However, the cost becomes a major issue when you do that on GPUs. This is where the TPU shines since you can scale out without cost crippling you.

I work on training on GPUs and the TPU is definitely an incredibly useful piece of hardware, not a vanity project.