You can simply join those channels. Getting an invite is not hard, or even unnecessary, from what I hear.

End-to-end encrypted means that the server doesn’t have access to the keys. When server does have access, they could read messages to filter them or give law enforcement access.