All contraception is very effective if used properly. Condoms have a generally 82% effectiveness [0] when improper use is figured in (98% if used perfectly).
But hey, the pill is also only 91% effective when improper use is taken into account. [1] So, really, relying on any birth control you don't control is crazy.
So your excuse for not using a condom isn't mathematically sound. Using it as the sole method is perhaps risky, but you control it. As a second-layer check against potential (unintentional) improper used contraception on the part of your partner, it still severely decreases your risk.
[0] https://www.plannedparenthood.org/learn/birth-control/condom...
[1] https://www.plannedparenthood.org/learn/birth-control/birth-...