That sounds like it would be dangerously MITM-able. Then any invite or link in matrix is equivalent to "let me log in as you" on some other random service.
Then I guess I'd have the backend send the user a link with an auth token after joining, that way at least no pasting needs to happen.