How long from now are you talking about? Eyeballing it, I think it would take us about a thousand years to get off by an hour?
However, in the past 50 years, we've only had 27. So at the current rate, probably more like 6667 years to be off by an hour.
Of course, if the earth's rotation changed speed by so much that 2 leap seconds per year wasn't enough, we'd find a way to cram more in. In fact, in that case, the case for leap seconds would probably get stronger.
How long has the 9-5 even existed? Less than 200 years? At no point in 6667 years would anyone notice any consequences of the drift, culture and work schedules change so much faster than that.
The Julian calendar being off with regards to leap years wasn't a problem… until it was. And then someone had to fix it: