This, in part, is why distributions like Debian go to the trouble they do to build everything from more-or-less isolated sourcecode artifacts.

Sure, its not always perfect, and getting say, blobless, is a challenge of its own, but no distro in their right mind for example, would ship arbitrary .o or ld.so objects in source packages.