How so? Hours, minutes, seconds, and milliseconds are all man-made units. It’s the failure of 24 hours not fitting exactly into a day, not the inverse.

Leap years have nothing to do with milliseconds in a day- it’s days that fail to fit into a year; days and years are defined by the orbit and rotation of the earth. Leap seconds, however, are another story...