You might be interested in this line of work: https://eng.uber.com/neural-networks-jpeg/

Training straight from DCT coefficients to avoid spending time learning a similar representation in the bottom layers of the net. I've personally toyed with something similar on GANs to gauge the computational benefits of not doing convolutions in the bottom layers of a net but learning directly in a FFT-like compressed space instead.