If all you do is download the release binary, anything could compromise you. If you read the source, an inauthentic release binary could compromise you. If you read the source and compile it, a compromised compiler could compromise you. If you read the compiler source, bootstrap compiling it from hardware code (reading all the source along the way), read the app source, and finally compile it using your compiled compiler, then compromised hardware could compromise you.

Every step along the way you increase the severity of the statement "If I'm fucked, then so is everyone using X". You stop when the set of people using X grows so large than you are sufficiently worthless in comparison.