I mean, there's a reason Banks chose to write Consider Phlebas with a narrative perspective from someone on the side of the Idrians. As I said in another comment, I think Banks fully intentionally means to make this perspective visible, legibible, understood as not entirely unreasonable.

Which is part of what makes the books so enjoyable to me, being invited to see multiple perspectives (especially reading Consider Phlebas after reading others that establish the Culture from it's own point of view).

Sure, Banks is portraying the best society he can think of for what he values and wants -- but acknowledging that even the best society he can think of has warts and can be seen by some as a dystopia too, and that not all might share the same values and wants.

I have to admit to a soft spot for the Affront. Terrible terrible people, but they did seem to be enjoying life more than anyone else.

Idirans were literally a genocidal fascist theocracy.