It's a huge pain in Matrix with even a small amount of users in a chat. At least for me, I've never used IRC where I cared about e2e. There are other services for that, including just encrypting with the other party's public pgp key.