Working conditions are also a factor.
Good luck finding a SCIF that's as pleasant as a home office.
But yeah, salary is a big issue. Defense work (whether as a civil servant or a contractor) just doesn't pay as well as VC-backed software development.
Although in the current job market, I have to say that the median pay gap seems to be smaller.