We already have a leap year, where we add an extra day every four years. So why could we not adjust the clock every 500 years when it gets out of sync by 20 minutes?
On the other hand, we already have leap seconds. They're not much of a problem.
Well, leap seconds are going away, so we don't.
They were never adequate for the long-term anyway, as the "two leap seconds a year" scheme will break down as the earth's rotation slows over the centuries and more and more leap seconds will be needed.
We're talking about an hour time drift over a few thousand years. Does that really matter? It's like caring what Julius Caesar's sense of "morning" or "evening" was. They never really solved any practical problem.