That's actually a super interesting point and problem! One of my favorite museum visits was "the museum of time" at Greenwich (of GMT fame). Well worth the visit!
A summary of the difficulty and "technology" of time + clocks.
1) Latitude (up+down) is easy to calculate + agree upon anywhere in the globe (but what does this have to do with time, you say?). To calculate latitude, you (mumble mumble) look at where the sun is at high noon, do some math and basically figure out if you're in a "small circle" (near the poles) or "big circle" (near the equator). Effectively, latitude is "defined" as "how big around is the ring you are currently on". https://commons.wvc.edu/rdawes/ASTR217/Gnomon.pdf
2) Conversely: longitude is nigh impossible to determine, and very few people think about why that might be. (and why it has huge commerce and military implications).
Why is it difficult to determine? ...well... what is "zero"? For the equator, it's well-defined. The biggest ring around, getting smaller as you go "north" or "south" (towards the poles).
For longitude... is it "how far away from mount everest?" "how far away from new york city?" "how far away from greenwich?" (Turns out, we've settled on the last definition).
Once you've gotten farther away from visible land (can't see the tallest tree/mountain any more) you don't know how far _away_ you are from anything (east-west) ... but you can trivially determine how north/south you are (look at the sun, do some math, figure out which "ring" you're on).
...but _wait_! You _CAN_ figure out how far _AWAY_ you are from something. If you "steal" the sun at high noon in greenwich, wait 24 "true clock hours" (in greenwich), and you'd expect the sun at high noon (12:00pm on your watch) to be exactly in the same spot. Go 100 miles west, and the sun would show up a little late (compared to your 100% accurate "true clock watch"). Go 1000 miles west, and it'll be quite a bit later until the sun shows up at exactly solar noon (overhead, casts no shadow). Figure out exactly "how late is the sun?" and bam, you've calculated "how far away you are from greenwich".
Summary: North/South is "easy" (how big around is the globe). East/West is "hard", and the first, best way to measure it was "how many hours east/west are we from greenwich".
Military and Commerce implications? Accurate maps. If you go west for a month, and run into a spot of land... you know your "Y" (north/south), but you may not know your "X (east/west). If you send 10 boats with 10 inaccurate clocks going "west", they may have wildly varying calculations as to what their "west" is, depending on how accurate their _clocks_ were!
So it turns out that if you ever really do get sent back in time, and happen to be wearing a decent watch, you could become godly rich simply by being able to produce maps that are accurate in the east-west direction and not just the north-south direction (since early on, you're competing with boats carrying grandfather clocks, or burning 24 candles that each last an hour) as a mechanism to keep track of when noon is when they left, so they can calculate how far east/west they've gone.
Spoiler alert, eventually somebody figured out some other fancy math involving maybe the moon or something in order to be able to calculate a longitude, but I believe that was after or near co-incidental with more accurate spring-wound clocks which reduced the importance of being able to do it simply via calculation.
Nowadays we take GPS for granted, but it's surprising how important of a problem it's actually solving.