It's not hard to demonstrate that apps are performing end-to-end encryption even if you don't have access to the source code. Reverse engineering this stuff is really pretty straightforward.
You can't just insert yourself in the message stream since the client and server use pinned, mutual certificate authentication. So you have to start from first-principles and step through decompiled code.
I'm not sure what you mean here. It's easy to identify where the key comes from and whether the ciphertext is what you'd expect it to be in that case.
> they aren't sending other data over unannounced side-channels.
It's not straightforward to determine that even if you do have the source - you could imagine an implementation that deliberately leaks information through timing details without that being obvious from the code. At some point you have to trust that authors aren't doing something awful.
> So you have to start from first-principles and step through decompiled code.
Well no, because the first thing you can do there is just disable certificate pinning. But really, the difficulty of stepping through decompiled code is vastly overrated.
As for using the correct key, dismantle the signal message envelope until you get your blob of encrypted message. Then see if the same blob appears on the target device. Multiple keys? I imagine either correlating message size and network traffic (encrypting stuff twice could well show up), or going at it with a debugger.
Which is really the answer to all of these questions instead of any network shenanigans. You root your phone and attach a debugger, then step through what signal is doing.
Not a security researcher, never reverse engineered anything for security reasons in my life.
Facebook distributes your 2MB pic to many people, does it technically require more than 2MB of your upload bandwidth? No. You only need to upload it once to their server.