It'd be a lot of trouble to interfere with the source, yes.

I think the release files is the place they could most easily tamper - generally they're stored on Github infra so the files could be changed, and the checksum on the download page also altered (or different files and different checksums provided to different people if targeted).

Unless the builds are totally reproducible it'd be tricky to catch.