Indeed! In fact, I have a project [0] from last year that uses a GPT-style transformer to address that exact issue :) However, it’s hard to go far outside simulations in real home robotics without a good platform, out of which efforts came Dobb-E.
I've also seen the one that uses the diffusion process for planning, I imagine it's even slower, but maybe with a consistency loss something can be done about it.