I was raised in California but have spent considerable time in areas of the US with 4 seasons (such as Chicago). I have come to love the 4 seasons for this exact reason. Sure, hot/humid summers are tough, and winters can get chilly, but I would keep them for the cyclical nature they produce. They make it so life continually changes, but more importantly, they make the passage of time explicit in everything (what you see in the city, what you do, how you dress, etc.).