"...every token embedding interacts with every other token embedding before it"

And, in the case of BERT, every token embedding after it too.