Of course, there's people like you for which it helps less than hurts but all large studies I can find suggest you are in the minority, not majority.
You're conflating caffeine with coffee, specifically coffees non-caffeine properties.
If you're referring to the Harvard study, that is coffee and its antioxidant properties, they even state [0] "and it doesn’t appear to matter if the coffee is caffeinated or decaf, brewed or instant."
The same antioxidants are widely available in many plant based foods and beyond, something lacking in an American diet, so coffee in that context is beneficial I guess.
However, the caffeine in itself does nothing to reduce mortality, its primary role in plants is an insecticide. In humans its classed as a neurotoxin.
caffeine has no health benefits, beyond cognitive, so sure it might stop you crashing your car if driving tired, but that's about the extent of it.
[0] https://www.hsph.harvard.edu/news/hsph-in-the-news/coffee-lo...
This recent meta-analysis reviews 40 studies.
> An inverse association between coffee consumption and all-cause mortality was maintained irrespective of age, overweight status, alcohol drinking, smoking status, and caffeine content of coffee.