You are right about some of the complications.
You'd want https://en.wikipedia.org/wiki/Reproducible_builds for a clear relation between source and artefacts. Then build-server hacking would be slightly less problematic: at least anyone would be able to check whether the server had produced any compromised builds.
And you could have multiple independent entities producing builds in parallel for your open source projects that need to agree. So a hacker would need to take control of all of them.
(You don't need to figure out byzantine fault tolerance or something like that: if there's any discrepancy, sound the alarm bells and have humans investigate.)
Every build server would sign their artefacts, but the signature would just certify that some specific sources compile to some specific artefact. Not that the sources are trustworthy.
Now you need to trust: the author of the sources of the packages at the times of signing, and that at least one of the all agreeing build servers that you ever collected signatures from for the package was not compromised at the time of signing.
That signing build server doesn't even have to be the same server that build your binary packages, the latter could be run by Satan for all you care.
Yes, in practice you will trust a central registry. The benefit of the signature model is that you don't have to trust the whole chain of provenance, but only at specific points in time.