Exactly. 20% or 40% sounds really big. In reality, you're maybe lowering your chance to die (during some time) by less than 2% by exercising almost constantly. Now, if you're not enjoying exercise, or feel exhausted after exercising, you actually reduce the amount of "quality life" by exercising.
If exercise was a drug, they'd never allow it on the market.