I agree! That's why remote attestation or simply verifiability is such an important feature of these schemes. Semantic attestation ofc means having access to the source code and that IMO makes open source the natural choice. Audits might be an option where OSS is not desired for other reasons (likely business related). Not an expert on FHE, but confidential computing provides that attestation feature and it's just a matter of the software to make use of it.