I am all for controlling CO2 and moving away from fossil fuels but I also agree with you.
Climate is complicated, there are many complicated feedback loops going on. As just one example warming temperatures promotes plant growth especially algae which captures CO2 interning back to the ocean floor. Of course some of that algae is not good for us (red tide).
I am not worried about life on earth, it will adapt and survive, it has seen worse, I doubt we would turn into Venus, but its a slight possibility.
The issue is what happens to human society, it will most likely adapt too, but at what cost? Rising sea levels, food supply having to adapt, these things could caused great suffering to many, then again maybe not. I don't want my kids to suffer or their kids.
The point is we don't exactly know how it will play out so why push it? Lets get off fossil fuels as fast as we can, we know its the right thing to do for many reasons, one of which is to stop returning so much CO2 back into the atmosphere.