"Live and let live" is a political philosophy. I don't see what is so hard about this concept: human beings typically don't agree on everything. They have to decide what to do. That process is called politics, whether it happens to result in a live and let live attitude towards many things or the oppressive politics of the Soviet Union. When you tell me that in many contexts people should not feel pressure to adhere to a specific political line you are, in fact, expressing a (non-universal) political ideal, one which you don't need to search much to find counter-ideologies to.

I happen to agree with you: my preference is one in which dogmatic and performative adherence to a specific ideology is discouraged but that doesn't make me non-political, especially if I happen to live in a society where (for example) religious fundamentalism is the main cultural artifact.

At a deeper level, assertions that we should "remain apolitical" are, in the end, simply assertions that the status quo is "sufficient" or even good and that people who have some difference of opinion about how human affairs are managed are "bothersome" for bringing it up. Sometimes this is motivated by a misapprehension or genuine (if incorrect, in my opinion) belief that the status quo reflects some obviously "correct" view of things and thus questioning it is pointless, bothersome, or even dangerous.

I think political differences should (for the most part) be tolerated, resolved through discussion and democratic processes, and that a healthy society is one in which people are (to the extent possible given other goals) allowed to live as they please, but I do not think of this as a non-political goal. I mean, just look around, plenty of people want to take that sort of society away from you.