As you point out, that the relative ordering remained stable means progress being made is not simply overfitting to the test set. The negative finding from the paper you link to is in how much accuracy drops considering how slight modifications to the test set were. In their own words:

> We view this gap as the result of a small distribution shift between the original CIFAR-10 dataset and our new test set. The fact that this gap is large, affects all models, and occurs despite our efforts to replicate the CIFAR-10 creation process is concerning.

> Nevertheless, the accuracy of all models drops by 4 - 15% and the relative increase in error rates is up to 3×. This indicates that current CIFAR-10 classifiers have difficulty generalizing to natural variations in image data.

It remains to be seen if others can think up ways to maintain performance that the authors did not manage.