It is possible if you think about it beforehand -- genrally you use a tone and/or a visual indicator to show that a transmission is in the clear (unencrypted). Other clients can then have a mechanism to allow speaking in the clear (sometimes requiring PTT -- press to talk). When encrypted transmission is ongoing, the gateway to the encrypted net may output a 'talk tone' that indicates the channel is busy. In this way a low grade client can join the conference, but they hear nothing but beeps (for a pure voice circuit) or see a busy indicator (for IP circuits) until someone wants to talk to them. When they talk, it is unencrypted till it reaches a gateway, after which it is either encrypted with the gateway key or passed unencrypted. (For radio broadcasts you just hear crackle or nothing.)

Obviously for un-trained users you often have the problem where someone transmits encrypted when they meant to talk in the clear, and vice versa. This is the reason for the 'plaintext' or 'ciphertext' side tones. If someone starts talking in the clear when they should not, another participant may choose to transmit over them, known as stepping on their transmission.

None of this requires there be a group key known by a central server.