Then that person is an evil asshole who should go to jail for purposefully infecting others with a disease, just the same as if he were to purposefully put anthrax spores on your pillow. However, he's not a rapist. That's something completely different.

Can someone agree to a blow job, but not penetrative sex? And is taking off a condom without prior agreement also fine?

Sure, that's a fair point. But there some might argue that that's a different kind of sex. Don't oversimplify my point: obviously agreeing to a blow job is not the same as agreeing to penetrative sex or agreeing to anal sex or agreeing to be tied up in a harness and whipped. But once it is clear that there is agreement for that kind of sex, you can't go after the fact and say "oh actually that was conditional on X and therefore you raped me".

As I said, the person who betrayed the other person's trust may still be an asshole - but a rapist, they're not. Let's reserve "rape" for what it actually is: a violent and despicable act of "taking" a form of sex that was refused.