> why would a highly qualified person without any preference towards civic duty subject themselves to all of this when they have their pick of jobs?
Well, all other things being equal they wouldn't. We cannot infer that those who are civic minded are more/less competent from that though.
In practice there are numerous reasons why highly talented people are willing to sacrifice from organizational flexibility and freedom of expenditure work in those organizations. The easy one is meaning. You don't have to be civic minded to get more out of helping cancer researchers get research data than making Accounting Excel Plugin v12.3. Variety and scope of challenge is another. Few companies can offer the kind of influence that being on a national standards org can. And since the gov can't really be salary competitive, it tends to compete on work/life balance - pensions, paid vacation, flexitime, sane office hours, no unpaid time on-call over weekends, etc...
For people like us though the answer is even clearer: cool toys. Oak Ridge National Laboratory works at scales so large I've literally had to google what the metric prefixes they were using meant as they were so huge as to be unfamiliar. Data centers, tools, software packages, etc chosen for cost and market fairness, and not just because some VPs golf buddy got wood from a slide deck...
--
Me, personally, like many here could be making roughly an extra car a year by working at RandomConsulting Co. I would wake up earlier, yell at my family more, be sitting in random office environments with smelly people constantly, lie-smiling while blowing smoke up customer asses in dumb meetings routinely, travelling 15 hours more every single week, be in airports all the time, and fundamentally trying to trick customers into building my resume while also being forced to follow their crappy tech choices and crappy tech environments...
Instead I'm building highly scalable open source, license free, BigData solutions on Kafka, Kubernetes, and Akka driven by F#. We're saving stupid money we get to use for schools and hospitals instead. I can tell my boss to suck an egg if I want, wear comfy clothes, and there are formal processes in place to protect my decision-making... Worth the financial "hardship" to have a pure, tech-driven, position, IoW.