Leaving aside the effect of prosody mentioned in another comment, I think the rest of it is habit, together with the brain's tendency to group things. If you're used to hearing "plus or minus", your brain may be grouping it together into a phrase whose meaning you understand directly without decomposition, so if you hear "minus or plus" there's a moment of having to map the components to the composite meaning, together with wondering if the speaker intends the difference to have a meaning.
(In mathematics, there's a reason to have both ± and ∓, since you can use them both in the same equation when you need "minus when the other one is plus, and plus when the other one is minus". For instance, a±b∓c means "a+b-c or a-b+c".)
I don't know if this has a name, but if it does I'd love to know it.