The builds are reproducible. Developers can also choose to fetch binaries directly from their repositories, and fdroid will verify that those binaries match the sources. So this is actually safer than downloading directly from developer's repo.

https://f-droid.org/en/docs/Reproducible_Builds