You could still make the phone a primary device and allow it to perform the key agreement and pass control off with a QR code, but that is complicated and leaves open the question of who is allowed in this conference.

So perhaps you just give up on persistent identity: just have an unencrypted waiting room, the organizer and their delegates can approve people in the waiting room to enter the encrypted conference.