I wouldn't trust any time unit larger than seconds if I was concerned about my units staying consistent. Hopefully time-keeping bodies don't decide we need leap-millis at some point.
Which is only to say that this is a problem that can't be solved simply by discarding some concepts that you find problematic. There's a lot of shit about timekeeping that's problematic. I'd advise tossing all the problematic shit when trying to figure out some internal representation for the underlying time that more or less always makes sense and then building out more human-usable representations with the uglyness on top of it.