I think it's due to it being essentially forced mindfulness meditation. Though you could swim with music, I don't and that means it's just my thoughts for 30-60 min (watch the thoughts, can't do anything about them right now). I'm forced to focus on breathing on a set schedule (analogous to vipassana techniques). I'm moving my whole body in a way that requires balance, complex coordination and attention to what my body is feeling (analogous to body-scans, yoga). If you're not listening to music it's also a little like sensory deprivation, there's just neutral water rushing sound.
I know running and biking can get there for some (myself included occasionally), but on a pool swim there's nothing to worry about other than the occasional turnaround at the wall. No obstacles, no dodging others on the sidewalk, no traffic. You're in a controlled, safe environment.
That or maybe it's just the hypoxia ;)