Cost Comparison of Deep Learning Hardware: Google TPUv2 vs. Nvidia Tesla V100

As is, it looks like p3 16xlarge spot instance (or probably preemptible gcp 8xV100) are still by far the most cost effective option. You have to do a bit extra to tolerate preemption, but it's worth the ~80% savings.

Also, the "TPU pod is 200x faster than single v100" comparison is a little goofy. Might as well say Summit or Titan is faster than my desktop.

All networks were trained with the same hyperparameters. Only the batch size was increased with the amount of parallelization used (so increased 8-fold for TPU and distributed GPU).

> Like they mention, they also use a very small network that isn't something you need the power of a TPU to train quickly and may scale differently than a large network.

I agree completely here. We unfortunately didn't have access to the TPUs long enough to create more useful benchmarks on networks like Resnet-50 and with bigger datasets.

> They also don't post their code so I can't check that their problems with ADAM aren't due to using L2 regularization, which https://arxiv.org/abs/1711.05101 shows leads to worse performance than SGD and you should use weight decay instead.

The code is the same for all devices and in the single-GPU and CPU benchmarks we see Adam performing better than SGD. We did not use any regularization besides a dropout layer so I don't think this explains the bad Adam performance on TPU.

I'll make some effort to clean up the code so it can be shared.

The article indicated they used the same implementation on all device types, so I don't think L2 regularization would explain why the results are fine on CPU+GPU and only bad on TPU.

I believe they are present in the Turing products from nVidia too.