The overwhelming concern is carbon that was newly put into the atmosphere from fossil fuel sources comprising millions of years' worth being released all at once (less than 200 years). Carbon that was recently removed from the atmosphere by plants which were subsequently consumed and reintroduced into the atmosphere is a very minor concern.
It's not that your perspective is wrong, just that it is dwarfed by many orders of magnitude, in respect to fossil fuels.