Something to consider: The use of timezones in mostly 1-hour increments over inconsistently placed areas means that the vast majority of people are already living many minutes off from the "actual" time at their precise location, in some cases even more than an hour. "Giving-up" implies that this is something important worth maintaining, where-as for the vast majority of people they gain nothing from leap seconds or even leap minutes. The most important thing for people is simply that everybody agrees on what time it is, which is easier when leap-Xs aren't done.
That said it's also probably true that a leap-hour would never actually happen, but that's not some big issue. By the time we got to the point that a leap-hour would make sense people would have already adjusted their habits and it probably wouldn't be worth it.