"Our findings demonstrate that pure 16-bit floating-point neural networks can achieve similar or even better performance than their mixed-precision and 32-bit counterparts." This is a very deceptive statement. Take 100 initialization states and train a FP16 vs a FP32 network, and you'll find FP32 will have an accuracy advantage. It's certainly possible to conclude this if a small sample of networks are trained. This paper goes on to state, "Lowering the precision of real numbers used for the neural network’s weights to fixed-point, as shown in [11], leads to a significant decrease in accuracy.", while later concluding, "we have shown that pure 16-bit networks can perform on par with, if not better than, mixed-precision and 32-bit networks in various image classification tasks." The results certainly do, but that doesn't really give an accurate evaluation of what's really going on here. A FP64 network can fall into a local minima and be outperformed by a PF16 network, but is it correct to say the FP16 network is better. I'm getting a lot of mixed signals.
I feel like, "significant implications" is quite a stretch.
A few concerns: Besides figure 3, other results do not provide side-by-side test vs validation accuracy to attempt demonstrate the network is not overfit, and the only mention of normalization was the custom batch normalization operation.
This may more be a rant about the current state of ML, but in a perfect world, we wouldn't use GPUs/would enforce deterministic calculations, results would be replicable, we'd train hundreds if not thousands of networks to draw conclusions from, we'd better understand how to visualize network accuracies and overfitting, and all datasets would be free of bias and accurately generalize the problem attempting to be modelled. We can dream.