If a package is easier to ship as a binary because the source is hard to distribute reliably, we give up hope of end users being able to modify and build their own versionsYou might be giving up hope too soon, because what you describe has in my experience been the life story of source code from the start, to some extent. And I'm not blaming it for that. Distributing source code (well, not the distribution itself, but getting it to build) on all possible systems out there is hard and impossible to get 100% correct. It's simply inevitable to encounter systems on which it doesn't build because of their specific configuration, or user error.
Despite that, end users have been building their own versions and still do. Users who really want to don't let themselves hold back by a compiler error. Granted it's possibly that in a perfect world where everything would just work there would be somewhat more users doing that, but there's no such thing as a perfect world.
Moreover I do have the impression things are actually much better these days, mainly thanks to CI which makes it easy to get out of the bubble and build on multiple platforms. At least I don't have that feeling anymore of 2 decades ago where I sort of was prepared to waste hours getting anything to build which got handed to me on the internet. Still, it would be very interesting to go and see the percentage of projects out there which just build according to the instructions.