1) He is allowing someone to diminish the voices of those who have actually suffered rape and need justice. He is also giving ammunition to those who attempt to make the argument that most purported victims of rape are "crying wolf".
2) He makes himself look culpable. This kind of thing is serious and if he's in the right he needs to make it known. The damage to his reputation is already done.
By standing up for himself he'd also be standing up for real victims. Engaging this kind of thing in the legal courts and the court of public opinion can't be easy. I hope at least there can be some resolution.