I think the point here is that all architectures can execute a certain type of program. An RNN maps to a finite state automaton, and this is an attempt at showing what the natural class of programs a transformer can implement is.
Turing completeness requires access to unlimited read/write memory. RNNs only have a fixed dimensional state. I guess I'm theory that starte is continuous, but it has to be a pretty optimistic model that assumes we can handle unbounded data like that.