> with process for turning source to binary, you need to trust compiler, cpu, flashing hardware and software and the whole lot of other things.

"We should not solve this solvable problem because other problems exist" is false.

Meanwhile the other problems have solutions, like reproducible builds, so that the attacker not only has to compromise your compiler/CPU/hardware, they also have to compromise any others the output result gets compared by, or one of them will differ and the attack will be detected.