Thankfully climate scientists don't just eyeball a single graph. And as a matter of fact, you conflate two things, one is that we know that now industrial emission is the cause for global warming and the other is, that in the past warming has lead to the release of CO2. These are two different things, and taken together only implies a positive feedback loop.

On the second question, the problem with global warming is, that everybody talks about the weather, and all our systems are well adapted to the current climate. Concretely, there are farmers at the edge of the Sahara, who are adapted to a semi-arid conditions, and when the Sahara moves south, then they have to deal with desert conditions. Similar, when the Great Barrier Reef moves south, then there are dive centers without a reef north of the new location.

So we will have to adapt to a warmer climate anyhow, but not cutting down emissions is an entirely unforced error, it makes global warming worse, without any concrete benefit. Cutting down emissions is precisely the "smaller fallout of manmade climate change" you are arguing for.