It's probably not coincidence that the former environment I spoke of was also a government contract.
Government work is hard for the reasons you mentioned, and also because generally, contracting pays well enough to price you out of other markets.
My exit strategy was to find a government vendor (e.g., that makes products for government / enterprise consumption) vs. a government contractor. The services side of the house is still pretty much a suit-and-tie affair, though we pride ourselves on being much more meritocratic than our customers, but on the engineering side, pretty much anything goes.
The one thing I will say about gov contracting is that as a result, I interfaced with much more than would have been in my normal comfort zone, and I came away from it MUCH more knowledgeable than I would have had I not done it at all, and the work was very rewarding. My advice is to hold out for the good parts as long as you can while quietly looking for a way out that seems fulfilling.