There is some talk of eliminating the leap second, which would over time have the Earth and sun diverge with regards to noon and such. One 'answer' to this concern is to have a 'leap hour' or something in the future (some future generation's problem, not ours): but given that people can't even get February 29th correct now, and it happens regularly, I don't see how a one-off event would be made to work. It'd be a huge coördination problem.
Just look at the introduction of the Gregorian calendar: it was slightly off since the time of Julius Caesar, but that minor error added up over time, to the point that to get the equinoxes/solstices back to where they 'should' be 10 days had to be removed with the Gregorian calendar. And because of politics (or a religious flavour) it took a long while for everyone to get on the same page.