A good middle ground would be an equivalent to F-Droid, where the developer can only submit the source code and the code is either compiled by F-Droid, or it is verified as a reproducible build.

https://f-droid.org/en/docs/Reproducible_Builds/

What arbitrary binary are you talking about? This is about an open source project.