Reproducible builds are an absolutely achievable way of adding trust to that particular step and reducing the overall attack surface.
We don't need to live in a world of two extremes, 'curl http:// | sudo bash' or hand-built silicon.
Honestly, I consider it lucky that the people who typically promote this nonsensical bullshit are probably not actually security engineers or researchers, or allowed to do security related work in any way -- because they would be terrible at it with a position like that. Actual security engineers understand that there are things called "threat models" and "tradeoffs" and you can, in fact, measurably improve security in meaningful ways for many systems.
You might as well just starve yourself to death because -- hey -- eating is pointless when you could get hit by a bus at any moment.
Why prevent your application from having SQL injections? Why not just dump all your customer records and private keys on pastebin? After all, the NSA can just steal those secrets from your computer with some Unicorn Magic, so clearly securing anything is actually pointless.