Not to be pedantic but the gotcha is that you can't know they're using the open source software as-is. If they run a hosted service or distribute binaries you won't know. Also with cryptography any change (diverging from the open source software) can have regressions.
If the client software is all that's ever supposed to see plaintext, being able to see source allows you to confirm that that is (probably) the case and then compile it yourself rather than trust that they haven't thrown an extra step in that backdoors it.
It just seems like such a chicken/egg problem. Where does the actual trust come from (like holy crap web certificates seems unbelievably broken).
Ultimately it seems like it's just impossible to be 100% for sure what is running on another persons server without having access to it. Which is unfortunate.