The fact that change in the abstract can be good or bad doesn't mean specific changes don't make things worse. But maybe more importantly, the
rate of change makes a huge difference. If change happens too rapidly, as human driven climate change seems to be, you don't have time to sufficiently adapt to take advantage of the new pros or offset the cons.
Climate change will make a lot of places less habitable while making some places more so. Great, it balances out, right? Except most people live in the places that are habitable now, and if the habitable areas change overnight (in relative terms), billions of people are going to suddenly be living in the wrong places. Natural climate change over tens of thousands of years would give populations plenty of time to migrate, as has happened in the past. Human driven climate change over 100 years will not.