I'm not arguing with your feeling. It's very valid. But, I've lived in the Northeast my whole life. If you live here long enough, you learn to discern between different types of cold. 20F with moisture in the air is awful, especially if there's a wind pushing it. The chill gets through the layers no matter how you dress. I think people from outside the region associate this type of cold with the whole Winter. It's not that way all the time.
20F with dry air and no wind is GORGEOUS. You can easily spend the day outside and, if you keep moving, wear only a long sleeve tee-shirt and long pants. (People who live here know enough to keep a sweatshirt or coat nearby for sundown.) Some of my most memorable outdoor experiences come in this type of weather where we'll be active by day and sit by a backyard fire at night.
The larger issues in the Northeast are the lack of sunlight and municipalities often don't handle the clean up of snow and ice well. My experience is that northern New England knows how to handle clean up, but southern New England is a deathtrap for walking and driving due to untreated ice and snow piled to the sides of already narrow roads.
A few years ago I travelled to Anaheim for a conference in Winter/early Spring. For the heck of it I took a short vacation in LA after the conference. For 3 days I walked around Silver Lake, Los Feliz, and Echo Park. My energy levels soared. I can understand why people like living in that climate. I'd love to experience living in it sometime.