No, it's not "possible enough" and I strongly suspect you fully realize that.
A backdoor doesn't need to be in a form of an IF statement or something comparably obvious and silly. It can be a weakly seeded PRNG that would allow a "determined party" to brute-force the key exchange in a reasonable time. That would take man-years to fish out from a binary, and that's without considering that you may (be forced to) distribute an altered binary on demand and to specific targets only.
So in the end all we have - realistically - is in fact just your word. There's no way for you to prove that you are trustworthy by pointing at some random binary. The only option is to distribute reproducible builds from an audited open source.