Re: your suggestion that whatever cannot be perceived does not make a difference to us, I care deeply about the future wellbeing of my family, including those years I won't see because I'll be long dead. I'd bet you do too.
The point is that we experience time as flowing in a certain direction and can't experience it any other way. Knowing that we continue to exist in some tiny sliver of spacetime doesn't change that fact, and so it is not very comforting, for easily understandable reasons.
If you're on your deathbed, with your conscious experience about to flicker and die, no one in the room is going to be comforted by you saying "well look at it this way, my 2 year old self still exists somewhere that you can't ever access and that I'll never return to, and even if I did, none of us would have any idea that this conversation was going to happen in 90 years".