Sounds weird that I can generalise like this but I've got 10+ years under the belt and I can say hand on heart that I've had maybe 3 "slightly difficult" clients in all this time. I've dipped out into the commercial world about once and almost instantly was bombarded with assholes, so retreated back into the place I know and love.
My figuring is this: the people who work for them aren't there to get rich - salaries are pretty crappy - they're there because they're fascinated in wonderful things and histories. So that endless bullshit drive to focus entirely on making vast sums of money is largely (not entirely!) removed. On the whole the sector is thoughtful, kind, considered. As a result (or because of!), it's a sector with excellent diversity, lots of working mums, lots of older people, lots of enthusiasm, lots of drive.
I'm never gunna get rich because museums aren't rich. But I do varied, fascinating work and the people I work with are almost always nice people. That to me is a huge thing and has positively affected my mental and working life.