I'll be frank: if we haven't progressed to the point that those offsets generally don't matter by those points in time, we've probably already failed as a species.
I say we drop leap seconds.
But they matter to some applications, and those people have arguments trying to decide which to use.
But yes: the ultimate solution if the world were to move to atomic/absolute clock synchronization instead of referencing the Earth's rotation is that every few hundred years we would need to adjust all the time zones by an hour to keep "noon" in the middle of the day. Given that we do this every decade or two anyway chasing DST vogues, I think this would be acceptable.
It is not so much that the actual times drift apart very fast, it is that the clock tries to corrupt the meaning of the word "day".
If leap seconds cause problems for your application, you probably should find another source of synchronisation signal.
Plenty of applications need to know a precise number of seconds (or milliseconds or nanoseconds) between the present and a past or future date/time. Leap seconds throw a wrench in that. Whether UTC should serve the role of an unaltered time source is indeed a matter of debate (and for good reason), but it's pretty clear that we do need an unaltered stream of seconds (upon which UTC - or an intermediate between UTC and various timezones should UTC fill this role as advocated - would base itself, as would any other applications that require said unaltered stream of seconds).
This isn't even mentioning more far-fetched things like space travel (at which point the non-constant rotational parameters of an arbitrary planetary body are neither relevant nor appropriate for system-wide/galaxy-wide/universal/multiversal timekeeping).
Also seriously, leap seconds suck as an offset because a second is a huge amount of time for many applications.