I agree with you, but it is being used as one strike against him. Given that there is evidence that he didn't even do it himself, it further proves how desperate the people who dislike him are to drag him through the mud.

You shouldn't. There's nothing wrong with it. Having an interest in the opposite sex is the most normal thing in the world and there's no reason why you shouldn't be able to state it.