1. When the mic is soft muted, information is getting sent to the conference provider which could leak information about private matters.
2. When the mic is not muted all information is definitely going to the conference provider in a way they can decrypt so they can mix it.
That is to say that when using most conference software, you have already granted then access to contents of the meeting.
If you can't trust them to not miss use information they get when the mic is off, then you also can't trust them when the mic is on.