Then maybe in a few thousand years when we're a full hour out we can think about having a leap hour.
Abolishing leap years, on the other hand, would cause the calendar to shift relative to the seasons by a notable amount during your lifetime.
If we even care, at that point. We might well have switched time keeping systems several times by then, seen massive improvements in artificial lighting and heating and such.
I think that's the general idea of "what's the point of leap seconds". By the time they actually matter, we're not sure they matter anymore.