Great summary! I also recently wrote a post digging into the internals of wav2vec with illustrations:

The Illustrated Wav2vec - https://jonathanbgn.com/2021/06/29/illustrated-wav2vec.html