Meanwhile, as people got used to the clock, the society became used to staying in synch at hour and sub-hour intervals. As better clocks were invented, synchronization became counted in minutes. This simplified things, allowing people to do more in the day and coordinate less, as they learned to rely on agreements to independently do things at specific time. Shops started posting opening hours, school started posting lesson plans, professionals started billing by minutes.
Clocks became widespread, and synchronized remotely - first from the church clock, then over the wire, over the radio, finally over the Internet. The loss of shared time reference at minute-granularity became rarer and rarer, until it became unheard of (except in excuses of those who failed to keep their commitments).
Losing a clock now would indeed throw society into disarray. If all clocks magically stopped tomorrow, many would starve and die before society figured out some way to restore time-keeping by proxy.
Clocks are a perfect example of the ratched effect of technology. A new technology at first helps you, or gives you advantage. As it spreads out more, it becomes relied on. Eventually, reaching ubiquity, it becomes a dependency, and everyone is assumed to be using it.
In a modern society, you cannot live without a properly synced clock anymore.
In the story, the baker was the only one left with that ability, as others left the skill behind when the clock became the authority.
This problem has been seen when missionaries, or whoever else, try and bring tribes into more modern civilization. They have trouble adapting to modern civilization, but the skill loss during that process makes it almost impossible to go back to the jungle.
When skills are lost due to modern technology, it isn’t so trivial to get them back. Look at how much time people have spent trying to figure out how stone was moved in ancient Egypt, for example.
Sunrise changes throughout the year. Sunrise in December and July are not the same thing.
To set the clock, they’d need a reference for the solar time to clock time for the current day.
I ran into this in Rome when I visited last year. A church there has a giant solar clock where the sun shows up on a line on the floor at noon (or so I was told). At noon, nothing happened. After thinking for a second, I realized I would need to find solar noon, and went to Wolfram Alpha to find that solar noon, in Rome, on that day, was 1:19pm.
The other option would be for the town to arbitrarily set the clock to solar noon one day, and just go with it, letting the drift be the drift. But the time would not be the same as it was before the reset. It could be an hour or more off.