>but the AIs slowly start to drift.

It's essentially the tower of babel. Each person will devolve to speak their own internal language only they understand. Each language will need to be encoded down to its meaning to be reinterpreted. None of us will know if the transformers are accurately decoding, or if the other person is accurately interpreting the decoding (which is arguably already a feature of human language without the computers in-between.)