I remember asking my ninth grade high school teacher that. I had just read a book, one of Rick Cook's wizardry books. A programmer fell into an alternate, fantasy magic-capable world, and time was tracked by the actual position of the sun. I thought it was a great idea and asked my teacher about it.
My science teacher immediately came back with objections that I didn't understand until a few years later. If you were investigating science, lots of things would be problematic.
Since then, I noticed some interesting things. Lots of pre-modern practices and rituals tracked solar noon. Allowing it to flex meant that there were greater alignment with the solstices and equinoxes. In a magic-capable fantasy world, tracking solar noon is sensible.
I also remember reading things like, how standardized time was a modern invention that created a particular view of the world. And then an article about that debate between Borges and Einstein. We remember Einstein, but not Borges. Borges, in his time, was the pre-eminant philosopher on time. He was influential enough that Einstein did not receive the Nobel Prize for his work on relatively.