IIRC Signal just has each group member send each group message to each recipient with the standard pair-wise encryption keys. It's the message's headers that lets the recipient know it's intended for the group and not the 1:1 group.
Additionally the key is regularly updated to provide some degree of perfect forward secrecy and avoid encrypting for people who left the group chat
I think it has senders encrypt messages with each room member's public key, rather than a single shared key. (At least, that's what the behavior I've seen suggests to me.)
Here's the spec, in case you want to comb through it:
https://spec.matrix.org/v1.11/client-server-api/#end-to-end-...
https://spec.matrix.org/v1.11/client-server-api/#mmegolmv1ae...
OLM is the public key encryption scheme, similar to the Signal Protocol. It is used to exchange room_key messages, but not the room messages itself.
MEGOLM as linked in the specification: https://gitlab.matrix.org/matrix-org/olm/blob/master/docs/me...