> Nobody does this, of course.
Every Linux distribution out there compiles everything in their repo by hand. If you use an AUR package of almost everything on Arch, you are building the software by hand locally from source. You can pull build scripts from launchpad, the Suse OBS, or almost every distros package repository (they all have automated build systems for everything, and you can do it all yourself).
Oh, and there is a little distro called Gentoo some people use where you literally build everything from source yourself.
So nobody builds open source software from source, besides the millions of people who do it all the time. Sure, it is a tiny fraction of total users, but there is a dramatic difference between "nobody" and "somebody" just like there is a dramatic difference in confidence between "were using this secure crypto, trust us, its there! its not just something else masquerading as secure crypto that we have backdoors all over!" versus "here is the source, go build it yourself if you don't trust us".
The value is not in everyone building from source themselves, just like my confidence in my software is not in auditing every line myself. It is in the general freedom of information on the security, and by offering the information to test it myself I will have inherently more trust in a project even if I never take advantage of that source availability myself - even if nobody ever does find exploits in the disclosed code, there is still trust a proprietary vendor can never have that I implicitly give free software projects because it is much more precarious to backdoor transparent software than a black box, and by just making the box transparent at all you add value.