And IMO this opens up the window for future GPUs to do even better. That said, when 9 TFLOPs is ~$1,100 (2 x GTX 980), I'm not too worried about memory usage in the long term.
Although, I guess the cutoff might have to be a bit higher than 8.23. Maybe the neuron activations would never exceed that range, but some intermediate computations could.
Supposedly the new Maxwell GPUs have some new instructions for working with half-precision floats ( https://developer.nvidia.com/sites/default/files/akamai/open... ). I wonder how complete this implementation is, because half-precision might be sufficient to train a convnet, and this would result in a significant speedup.
You also cant clip them and get away with it, you have to either renormalize the layers to do half-precision (and live with the extra cost) or stick to full-precision. I was doing fun stuff early this year that did fixed-precision nets (8-bit/16-bit). Things get very interesting :)
http://en.wikipedia.org/wiki/Half-precision_floating-point_f...
That's plenty IMO for most inputs and weights. Where it gets tricky is in accumulation. You could constrain the weights for each unit I guess, but this is the sort of work best done under the hood rather than by the data scientist IMO. I'd personally choose 32-bit accumulation just because it would drastically simplify code development.
I've also worked with fixed precision elsewhere. It's awesome if you understand the dynamic range of your application. It's a migraine headache if you don't.
IMO What remains to be seen is what one could do with FP16 data mixed with FP32 accumulation. My signal processing friends think a pure FP16 network is just asking for trouble and are of the opinion that a minimum of 20 bits is needed for stable accumulation and 24 is more than enough. That's not far off from Teradeep's 8-bit data plus 16-bit accumulators for image data.