Well, there is a whole seperate line of research concerning the topic of these input perturbations ranging from PGD to just Gaussian noise. This model does not claim to defend against any of those.
Author here. I ran some early experiments a while ago, and it looked like adversarial examples for convnet classifiers didn't transfer to transformer classifiers and vice versa. Definitely worth looking more into!
Could you elaborate a bit more? What were the differences between transformer adversarial examples and cnn adversarial examples?
I didn't notice any obvious visual differences, but I'm also not an expert on adversarial examples. The transformer models were similarly susceptible to attacks, but while adversarial examples transferred well within a model class (~40%), they did not across model classes (~5%). These are rough numbers from memory, don't hold me accountable!