Montreal has even more brutal cold, yet has a bustling downtown worth walking around -- because it was designed for people.
Another anecdote to refute your point: LA has the best weather in the world, but a completely dead downtown that most people avoid.