chrony[1] neatly solves these and a host of related issues by guaranteeing time increases monotonically on a given host, speeding up or slowing down the clock appropriately. It's a nicer alternative to ntpd. Also, AWS recommends it if you are using their Time Sync service (which is GPS-locked atomic clocks in every region with leap second smearing).

[1]