They don't have to (and shouldn't) retain highly skilled mathematicians. Nobody is suggesting that everyone design their own ciphers, authenticated key exchanges, signature schemes, and secure transports. Peer review is good; vital; an absolute requirement. Committee-based selection processes are what's problematic.