Safety concerns?
E.g. these authors find the health improvement statistically increases your life expectancy by up to 14 months, while traffic accidents statistically reduce it by up to 9 days. That's a ratio of 47 to 1.
Edit regarding your comment: Oh, I see why you're confused. I didn't need to do the math because I was talking about a 1-time bike ride as an example to just get the point across -- it's already scary for 1 ride. But if you want the actual math for a lifetime, there's a ~1/5000 lifetime odds of dying in 1 year of biking. That's still pretty damn high. I don't know about you but I'd rather just give up on the 78th expected year of my life and lose the 0.1% chance of dying in the next 5 years.
But the point of bringing up the statistic is to check "is this risk worth worrying about, to the extent that I'm not gonna do thing X"? That's what a rational person does in all situations - is the risk of flying so high that I shouldn't go on holiday? No. Is the risk of falling if I climb that cellphone tower so high that it's not worth it for the view? Yes.
I'd take a 1/1000000 chance. I wouldn't take a 1/1000 chance. Get the point?