They use Whisper Systems specs:

* https://whispersystems.org/docs/specifications/x3dh/

* https://whispersystems.org/docs/specifications/doubleratchet...

I believe it is pairwise (i.e. every chat member does a handshake with every other chat member), although there is some debate about this in the academic community.

Does whatsapp do multiparty encryption? I didn't think they did.