You aren't being neutral. Your position is a very strong affirmative for the current status quo. You don't want anyone to be able to enact change.
That's an understandable position, but it isn't neutrality. Especially not if the status quo is actively negative for certain people.
As an aside, this portion of your statement:
>It's saying: not only is it not okay to disagree with me and be on the other side, it's also not okay to not passionately agree with my exact side.
Feels like a strawman. The existence of critique isn't a censure. It's just how rational analysis works. You find ideas and you work through them.
If your goal is to never be critiqued because you can't stand to be wrong, and that's why you've adopted a 'centrist' standpoint to get above it all that's a very political position to adopt.