He was accused of two things. One was removing the condom during sex and that is not seen as rape but a lesser charge but the other one, having sex with a sleeping woman without a condom was rape. He is obviously not convicted yet so we don't know how it would hold up but in Sweden if someone is too drunk, sleeping or in a otherwise vulnerable position it is seen as rape. There is some more nuance to it but that is generally so.
Of course women makes up shit but it seems pretty far fetched that CIA would plan a couple of fan girls to have sex with him and then claim that he removed the condom.