Yes, a hidden factor seems far more likely. There is probably some secondary variable that affects both your mortality rate and your perception of your own fitness. This seems much more likely, even to the point of seeming obvious.