Interesting, in their benchmarks it shows that Tensorflow is nearly twice as slow to train on a GPU compared to PyTorch. That's a surprisingly large difference.
2X slower is unlikely, but Pytorch does tend to be faster than TF in most recent benchmarks. See this link posted by another commenter for one example: