This is true, but at least it lets a small group of people do the verification work and post in a public place if the publicly distributed binary stops matching up with the one built from source.
I supposed you could pull apart the container format (apk or ipa) and compare the .class files (Assuming Java, I haven't looked at this software so I don't know if it is standard Android or a lot of NDK stuff) or ObjC object files one by one to look for discrepancies versus a local build using the same tools... hopefully someone volunteers to do that and keeps doing it again on each new release.
https://github.com/WhisperSystems/TextSecure/issues/127
For iOS I believe that decrypting the binary and doing an objdump, then comparing the resulting assembly is a reasonable approach to ensuring that two builds do the same thing. Comparing objdump results won't protect against particularly insidious backdoors like those injected through data resources or binary headers, but in tandem with a source audit should give a fairly respectable degree of assurance.
This process would be quite easy to automate.
Not a chance.
And if someone is doing this, we are well past "particularly insidious".
I imagine certain organisations knew about buffer overflow bugs long before they were used publicly, so imagine if this was the 70's and you saw some strcpy calls peppered into some useful code, would you really be able to know 1) the class of attack exists and 2) if it was intentional or not?