Assessing accuracy? If you're running the same architecture with the same training regime, why would you expect differing accuracy numbers (other than a bug of some sort)? Seems a bit strange to include.
> TensorFlow shared the training script for Inception V3, and offered pre-trained models to download. However, it is difficult to retrain the model and achieve the same accuracy, because that requires additional understanding of details such as data pre-processing and augmentation. The best accuracy that were achieved by a third party (Keras in this case) is about 0.6% worse that what the original paper reported. Researchers in the CNTK team worked hard and were able to train a CNTK Inception V3 model with 5.972% top 5 error, even better than the original paper reported!
This suggests that an improvement in accuracy is possible by switching to CNTK, hence why I included an accuracy metric from both frameworks. (and also for sanity checking, as you note)
But it's worth noting that this code is all released:
https://github.com/tensorflow/models/blob/master/inception/i...
It may be hard to replicate that across all platforms, though -- as an example, the distortions include using four different image resizing algorithms.
Some of it was true preprocessing, i.e., cleaning up the imagenet data. I wrote a bit about that here: https://da-data.blogspot.com/2016/02/cleaning-imagenet-datas...
(tl;dr - there are some invalid images and bboxes, etc., and some papers chose to deal with the "blacklisted" images differently.)
See for example the TF vs Torch implementations of Pix2Pix https://github.com/affinelayer/pix2pix-tensorflow (at the bottom of the page).
The reasons for this are many: different order of operations, failure to propagate random seeds, different order of processing files etc. I don't think there is a structured study of this so that would be a great thing to do!