Not really. For firefox anyone can build from source (see also iceweasel), disable automatic updates. For chrome it's mostly the same, but then for Chromium instead.
Firefox is giant. It shouldn't be hard for a malicious party — should one appear someday — to hide some tiny backdoor somewhere in a more-than-a-hundred-megabyte source code tarball.
Verifying GPG signatures of the tarball could prevent some (but not all) issues, but from my observations it's rarely done. And when I've seen it done public key's origin wasn't thoroughly verified, just blindly `gpg --recv-keys`'d from keyserver.