Binary neural networks, where weights and/or activations are just 0/1s, are an active research area. In theory they could be implemented very efficiently in hardware. But in contrast to FP16 (or to some extent, int8), just quantizing FP32 to 1 bit doesn't work very well. There have been successful methods in practice. There was a company called Xnor.ai that was built partially around this technology, but it was sold to Apple a couple years ago. I don't know what's the current SOTA in this area, though.