You make a great point. Health should be the topmost priority and I need to get better at it.
However to my original point - you can lose fitness (within reasonable limits) and then gain it back and not notice a difference later on. However time lost is lost forever.