That's correct. You can't mix without decrypting. Signal does not mix at the server because it can't.

More info is available at my blog post about it: https://signal.org/blog/how-to-build-encrypted-group-calls/