You can validate the code that's running on the client and the server, in theory
If Signal works well, you shouldn't need to validate what code is running on the server in the first place.
on the signal.org website, there is a direct download link right below the google play button.
it's still just a binary, but you can control updates. incidentally i just did an update this way.
If you reject that on the basis of "we can't know if it's what they're running" or "it's a partial dump", then I don't see how Matrix is any different. Not only we can't know if Matrix servers have modified software, but we also have to trust/verify several servers instead of a single one.