Something crazy I've experience is when I swam in an outdoors pool every morning early for a while. This involves several temperature shocks, like jumping into the shower (hot), then walking from the shower to the pool (cold), then jumping into the pool (very cold initially, then ok), then walking back to the shower then showering again. This whole routine happens over just 60 minutes.
Other than the overall health and wellness I got I normally associated with exercise, and improved breathing because of swimming, I felt immune to temperature discomfort! After a while, the routine itself didn't bother, nor walking 15 minutes from the parking lot to my office under 50C summer sun at noon. I could go out in winter in any clothes I want.
It was very noticable and very specific. Sensitivity to temperature itself may be a form of unfitness.