I understood the reasoning to be that to increase the range of accurately representable values in a circuit, you either need to increase the voltage or current used in an analog circuit (to achieve a certain accuracy versus a noise baseline), or devote more bits in a digital circuit. The first gives a linear dependence (or quadratic for I^2 losses) of power on range, the second logarithmic.
To give a bit of a dramatic illustration, if you circuit has on the order of 1 nV of thermal noise and you wanted to do the linear analog equivalent of 64bit integer arithmetic, you would need a signal on the order of 10,000,000,000 V to have enough precision. In fact, in terms of power consumption it's even worse. If the 1 nV signal consumes something like 1 pW, you would need something like the total power output of the Sun (on the order of 10^26 W) -- a bit of an expensive multiplication, no :) ? That's how crazy it is!
Again, if you can get away with less than 8 bits of precision and imperfect linearity the picture changes, but I wouldn't declare it superior a priori without looking at the numbers.
But yes, I understand your point. Both analog and digital implementations have their strengths and weaknesses. If you value power over precision, go with analog. If the opposite - go with digital.
The reason NNs don't exhibit strong error propagation is because of the non-linearities between linear layers that perform operations analogous to threshold/majority voting or the like, which have error correction properties.
Be careful with jumping to conclusions: I never even cited ReLUs or Sigmoids in my post! I don't have any opinion on which non-linearity is better, I only know both are dramatic non-linearities. My claims were about linear circuits. You should use whatever nonlinear element works best in your Neural Network, of course (and I've heard ReLUs have good advantages).