Do you also trust your package maintainer to not mismerge patches and leave you open to security bugs? Your argument is no bueno.
Additionally, it's fairly ironic this is about a browser. If you don't trust packages maintainers, yet you want to use a browser, which the whole point of is to download and interpret text, code and binaries which you have little in the way of actually controlling after pointing it at a site, then I think you've made some interesting security trade-offs in your mind.
In the end you will always arrive at a chicken and egg situation, you will ultimately need to trust the engineers who designed your CPU and chipset, the VLSI design software which they used, the developers who wrote the compiler and toolchain, the tools used to bootstrap it, external libraries, etc.
The world ultimately runs on trust, no matter how you slice it.
http://wccftech.com/intel-possibly-amd-chips-permanent-backd...
Really, this is what everything in life is like. Every time you cross a bridge, you are implicitly trusting the builders who built it, the engineers who designed it, the mechanical engineering processes they used, and the mathematical disciplines that they rely on, all the way down to their fundamental axioms. You have to extend trust at some point there as well, otherwise you can start by proving there exists a class of numbers we will call integers...
But of course nothing beats compiling from source.
Did you assemble a bootstrap compiler yourself? Your binary compiler could be backdoored! [1]
[1] https://en.wikipedia.org/wiki/Backdoor_(computing)#Compiler_...