yup, training on DCT representation in general is an untapped research direction. can also help with larger images, longer audio, etc
DCT is a linear transformation so the model ought to be able to learn it (and typically already does). If you're going to feed in some non-learned transformation it seems like it should be nonlinear to add value.
True, it goes against the spirit of deep learning
Still the Transframer results are interesting and has encouraged me to look into it