If I understand correctly sigstore can guarantee that the software is what the the author intended to, but it cannot guarantee that it is was built from a specific source code. It is somewhat like web site certificates, you can register and have a valid certificate for "facenook.com" (someone actually did) but that doesn't mean that the web site is credible.

It is really hard to verify that a binary is based on the same source code, in addition libraries and build environment are not always included in the repository itself.

If you build the binary locally chances are that the binary will be slightly different, due to changes in the build environment for example. You will need to do a binary comparison and understand the reason and meaning of each change.