The majority of the power in a smartphone is used to light up the screen. If your phone is in your pocket, it's a reasonable assumption that the screen is turned off.

It's not impossible, but the phone would have to be quite inefficient to have any measurable effect. Laptops are probably a bigger problem if you keep them in your lap regularly, since in that scenario they do operate at full power close to your scrotum.