Just a note, but it is possible for anyone to become hardened to cold weather with gradual exposure over time. I used to feel cold like death but now I easily walk around in casual clothes in 30 degree weather.
You are correct, and it's kind of insane to me how much of a "baseline" there is to what "tolerable temperatures" feel like. You start to notice this a lot living in a place with radical temperature swings. If gets up to 35F in February, I can wear a T-shirt. When it first drops to 45F in October, it feels like the arctic.
I'll happily go outside down to -5 C/20 F weather outside wearing shorts and a warm jacket, as long as it isn't too windy.