You know without doubt that the binary that you produced was built from the source code, because you built it.
I'm asking, why not stop there? You have a binary that you can safely consider "verified", what does it matter if it's bit-for-bit identical to any other binaries?
> I'm asking, why not stop there?
You do not know that it was built from the source code. That's precisely why you're not stopping there. For your machine, the one you're building from, may be compromised and compiling something else than the source code it's showing you (for example it may be inserting a backdoor here or there in the binary).
While if you really end up with the same build as everybody else does (and you have to really make sure of that: checking the hash from the very machine you compiled from is not be sufficient), you know for sure that the build is the same everybody else ended up with.
I encountered it when I updated a bunch of NixOS packages at once. Since the crash happened in chromium, I figured I'd downgrade that specific package by pinning an older channel. It still didn't work. But I knew that chromium had worked before, and nix doesn't really use the FHS, so the system wouldn't be polluted with random versions of packages. I also knew that the SHA for chromium had to match what was here: https://github.com/NixOS/nixpkgs/commit/a1c5e5bc40749652503c....
So if chromium wasn't the issue, the problem would have to lie elsewhere. Comparing the last two generations of my system, I found that Sway had updated, which also brought a new version of wlroots that broke older versions of chromium. I eventually found my way to this issue and rolled Sway back until the fix landed in stable.
There were probably easier/quicker ways to reach the same conclusion I did. But I think that knowing the software on my system was built correctly, and was relatively isolated (i.e. wasn't depending on random versions of libraries in the FHS) made it so I wouldn't fall down a rabbit hole that lead to a dead end.