This is very interesting. I have been wondering about this for years. I have asked neural network people whether it is possible to build a neural network from the non-linearity induced by rounding. I have never received a convincing answer. Would you be able to point me towards a write-up of this fact? And why is this not used in practice?

I imagine that how well this works also strongly depends on the kinds of rounding. I imagine that stochastic rounding, or the rounding used in Google's bfloat16 are different in this regard in comparison with standard IEEE floating point rounding.