I like to think I can stand the cold relatively well, but what always gets me is my ears. The rest of my body will be fine but my ears start hurting in minutes, which then translates into quite the headache.
I have the same issue with my ears in cold weather. They start to hurt pretty quick in cold temps, so I try to always either have some small wrap around ear muffs or a beanie. As long as I keep my ears warm and out of cold wind, I'm able to stay comfortable.