I don’t think I’m convinced of that. Most fitness measures saturate as you approach a limit. For example, you can build muscle mass rapidly in the first couple of years but it slows dramatically as you approach your genetic limit. It only takes a few years of consistent training to get to that asymptotic level. So on that measure, you can basically “catch up” to your hypothetical “always trained” self.

While you might not be able to get to the exact same level of fitness as if you had trained for years, I think you can get fairly close. And without a doubt, you can drastically improve from wherever you’re starting.