To be fair, overheating your scrotum will in fact mess up your fertility. The need for temperature regulation is the whole reason we have them in the first place.
It's not impossible, but the phone would have to be quite inefficient to have any measurable effect. Laptops are probably a bigger problem if you keep them in your lap regularly, since in that scenario they do operate at full power close to your scrotum.