Correct, but if there is something between you and other user and can intercept key exchange then it can decrypt and encrypt anything on the fly. I think you would have to exchange keys offline to have true e2e experience.