I'm mostly working on fairly simple image segmentation tasks but in my experience just replacing convolutional layers with attention layers + position embeddings works well. Using convolutional embeddings before the transformer encoder also helps.

It still take a lot more epochs to train though, so you might have to decrease the learning rate of your discriminator by a lot.