> I could just as easily say...

And I think you're right! It seems very likely to me that that happens.

Playing devil's advocate for myself, however: the comment was not uncharitable because it did not attribute intent. Someone saying "let's keep politics out of it" may be trying to avoid conflict or simply be respectful, but unknowingly in effect quashing challenges to the status quo.

The comment was meant to be an observation of unwitting outcomes rather than secret intentions, but maybe that's in fact what you're objecting to ("it's easy to be pessimistic about the aggregate effects of small, inoccuous actions, and that attitude deserves some pushback").