Transformers for Image Recognition at Scale [video]

I spent the weekend reading this super interesting paper "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", and made this video to explain it a bit. I hope it can be helpful for those who're also interested in how the transformer architecture (de-facto model in NLP) can be applied in computer vision.

Another video which explained also quite well the intuition behind this paper, from Yannick Kilcher: https://www.youtube.com/watch?v=TrdevFK_am4

Link to paper for convenience: https://openreview.net/pdf?id=YicbFdNTTy

Thanks for this super detailed video! I've seen some of your others in the past and always found them helpful. Yannick's video on this paper is great as well.

If anyone is looking for a written walkthrough, I did a quick summary/explainer on my personal site: https://whoisnnamdi.com/transformers-image-recognition/

It's still an interesting paper, but I was disappointed they were "just" concatenating an imagine generator with a language model. I'm really excited for when someone figures out concurrently trained models, say, alternating between training passes of GPT-3 and iGPT, such that the very same attention layers deal with both language and and visual/spatial conceptualization. I expect common sense reasoning capabilities to take a huge leap, at that point.

All these transformer explainqtions are missing a level of detail.

I wish someone would code this model and walk through it along with the training and inference procedures.