Curious bout your use case for building all software from source, is it because you're worried about the supply chain since nixpkgs builds don't have reproducibility guarantee?
So far, I'd say the biggest negative (aside from the build times that I was already experiencing due to the optimizations) is that GNU savannah will temporarily IP ban you when you download too much from them. For example, building the grub that is used for the ISOs downloads like 70+ patches from GNU Savannah which frequently triggers the IP ban.
Building from stage1 with customized CFLAGS was all the rage then…
Do you not trust hydra, the infra hosts, the people, all of it? What could be done to improve the cache's posture?
It seems the current situation is there is a single signing key that has been in use since 2015 which implies that key is not tied to specific hardware and therefore could be leaked/exfiltrated/copied: https://github.com/NixOS/rfcs/pull/149/files#diff-ac4cc42774...
Have you measured the gains from your source-based approach? I was into Gentoo for tuning a long time ago when hardware wasn't as powerful, but I hadn't considered optimizing at that level with nix.