Agreed, the transformer's approach to memory/attention is certainly novel and a non-trivial departure from RNNs. It's also questionable the extent to which this is not over-fitting. Some are already finding phrases on the web the model is clearly stitching together. This is not to underplay the impressiveness of stitching the correct phrases and n-grams in a thematically coherent way for paragraphs at a time, it's still very impressive (this transformer should be nicknamed Bumblebee).

Rather than under-fitting, I'd consider this diminishing returns in the model's ability to make use of the available information. You can get within 10% of it on the challenging winograd schema task using only 2%-5% of the used data. Many here are programmers and computer scientists, rather than stoop to hyperbole we should try to understand. What class of automata can this model learn? Finite state machines, push down automata? Transformer has memory but no loops so I very strongly doubt them to be more than that and yet they perform so well on language (but then again, not simple program learning) tasks.