This—improving trust in the system and making it provable—is something we're looking to do. Having the publisher sign the hashes and publish the public key is one approach we've discussed. You'd be able to validate this out-of-band. Most users, however, likely won't go to this extent - a lot of the motivation behind this work was around automating the stuff that folks just don't do (like comparing hashes).
Longer term we'd like to get more of these capabilities into the browser itself - comparing the hash in a separate context, validating signatures, etc - so that this is scalable.
Ultimately these systems are also still built on trust: at some level there are humans in the loop, an assumption that the user's machine itself isn't compromised, and/or that the code itself is actually "correct" (for some definition of correct).