You have to trust everyone who claims the binary is clean, yes. But the vendor must also trust all these conspirators! And they can't bribe everyone - it only takes one honest guy with a decompiler and it's game over.
I imagine certain organisations knew about buffer overflow bugs long before they were used publicly, so imagine if this was the 70's and you saw some strcpy calls peppered into some useful code, would you really be able to know 1) the class of attack exists and 2) if it was intentional or not?