That'd be because they are the only bitwise operations under which integers form an Abelian group.
Specifically, AND and OR lack an inverse operation.
By the way, while one indeed needs the ability to "cancel out"/invert, we can derive similar swaps in weaker algebraic structures (and don't need full Abelian groups). For example, non-zero rationals can be divide-swapped: a = a / b; b = a / (1 / b); a = b / a; or similar for subtraction-swap. In general, a quasigroup seems to suffice: a = a * b; b = a / b; a = b \ a (where * is the group operation and / and \ are the right and left divisions, respectively).