I haven't personally experienced something like this, but I can come up with a few reasons.

1) maybe I just don't want to switch seats. Maybe I'm already settled in, maybe I like the fact that it's a window/middle/isle seat, etc.

2) imagine someone refuses to sit next to you, stating the reason as "that person is ugly" or "that person's choice of wardrobe disturbs me". If that happened to me, I'd certainly be offended and might refuse to move out of principle (though personally, I'd rather the flight was on time than make a statement). Those reasons are different from "I can't sit next to a woman", but they also refer to superficial, and frankly, childish, things.