> Desktop Nvidia simply converts them to 32-bit floats up front so the performance is kept but memory is wasted

Pascal has native FP16 operations and can execute 2 FP16's at once ( https://docs.nvidia.com/cuda/pascal-tuning-guide/index.html#... )

BUT, and this is where things get fucked up, Nvidia then neutered that in the GeForce lineup because market segmentation. In fact, it's slower than FP32 operations: "GTX 1080’s FP16 instruction rate is 1/128th its FP32 instruction rate" https://www.anandtech.com/show/10325/the-nvidia-geforce-gtx-...