Even if you're fine with human civilization dying, this reasoning isn't right. It is possible that some processes could be started that cause positive feedback loops that eventually do hurt the planet as a whole, and Earth could eventually look like Mars. If you include the biosphere in your definition of the planet, it is already very much hurting. Think of overfishing, the species we've led to extinction, places where people have caused desertification, and so on.
Yes, the rocks will still be here. I don't think that means the planet is fine.