Interestingly, Transformer has already used residual connections before. The new part here is to actually not use the standard residual connections anymore but normalize after every layer (including the residual connection), and also have a factor on the residual connection. So you don't have a direct pathway anymore.

The argument is that otherwise, the gradient magnitude for the lower layers becomes too big. Which intuitively makes sense, because due to the residual connections, all error signals from the upper layers will end up at the lower layers.

I wonder why this is apparently not a problem for ResNet.

I hope so, but I'm not convinced. The experimental results don't persuade me that deep transformers are better than the existing extremely wide transformers with tons of parameters. Yes, they show a much better result on the multilingual benchmark (M2M-100), but that's not a super well studied problem, so I don't know that the baseline they're beating is very well optimized. And on OPUS-100 they only beat the baseline by building models with vastly more parameters, which shouldn't really surprise anybody.

What’s that?