Inference (forward pass) in convnets is really cheap even in floats, there are some awesome hardware like NVidia Jetson and most deep learning framework (i.e. Caffe) support it right out of the box. So we were doing realtime inference in embedded for years now, what am I missing?