So the downside is “you can more effectively yield your power here”, or more specifically: “you can take more in taxes and conscription.”
Centralized time might be pretty small in the scheme of centralization - and come with numerous other benefits - but, that generalized resistance doesn’t come from nowhere, or irrationality, or stupidity.
IMHO it’s not that “don’t tell me what to do” is baked into the human condition, it’s that we bake into our condition that when people tell you what to do, it’s better for them and not necessarily better for you.