You're probably reducing your life expectancy much more by skipping the gym (or publicly socializing, but that's a whole other discussion) than you would be from those 3-4 illnesses a year.
Regardless, I hear what you’re saying, but that didn’t sound like what GP meant when they said they missed their gym life.