I've had that happen to me a couple of times too. One of the most entertaining times was when someone refused to move from my window seat was because they wanted to "look out from the window". After I showed her my ticket stub, she just looked at me blankly. Then I explained to her that she is sitting in the wrong seat to which she replied that she doesn't understand.

I did relent because the flight attendant asked me nicely if I mind just taking the middle seat. It was the first time I got a window seat that wasn't over the wing!