I don't really get what the proposed alternative is here. To just let the time of day drift away from the time on the clock?
Then way not abolish leap years as well?
Then way not abolish leap years as well?
Then maybe in a few thousand years when we're a full hour out we can think about having a leap hour.
Abolishing leap years, on the other hand, would cause the calendar to shift relative to the seasons by a notable amount during your lifetime.
If we even care, at that point. We might well have switched time keeping systems several times by then, seen massive improvements in artificial lighting and heating and such.
I think that's the general idea of "what's the point of leap seconds". By the time they actually matter, we're not sure they matter anymore.
Religion.
For example, the various Christian groups want to keep Easter at the Correct Time each year, and the leap year is part of the complex calculations they use to accomplish that.
https://en.wikipedia.org/wiki/Computus It really is a bit involved.
I suppose they could redo everything to accommodate a calendar without a leap day, but the Julian to Gregorian update was divisive enough, and you really expect them to go through all that again so soon?
More to the point, calendars are more deeply embedded in the culture than things like TAI and UTC and so on. Changing a calendar is a maybe-once-in-a-millennia event.
And arguably the situation is similar for leap seconds as well. No one cares because of how far out substantial drift is. By the time it adds up to an hour, people will be used to it, or just shift their timezones (which happens all the time).