My understanding of both this and Apple AFT is that they are trained with attention, but then inference is done as an RNN.

Is your understanding different?