> earlier or later

My understanding was that leap seconds are only ever added. It’s not like this averages out over time. The problem is that because they are only added the effect is cumulative and eventually some future generation will be eating lunch at 00:00:00.