There are two distinct things going on here: formally "swapping" the arguments in the wavefunction, and physically moving the particles around. A priori, there is no reason the phase shifts from these two actions should be related.
You're focusing on the swap, but it's not interesting: your argument that the swap phase squares to 1 applies to both 2D and 3D equally well. When people talk about anyons, they're talking about the phase from moving the particles. That has more options in 2D because the paths one can take when moving particles around can have a greater range of topologies (https://en.wikipedia.org/wiki/Braid_group).
The reason that textbooks mix these things up is because in relativistic quantum field theories these two phases are always the same, by virtue of the spin-statistics theorem. But the anyons of condensed matter are nonrelativistic.