We know our bodies do follow circadian rhythms, so you have the initial link to length of day. Then everything else is split into sets of 12, a number with exceptional properties that natural selection may optimize towards:
"The number twelve, a superior highly composite number, is the smallest number with four non-trivial factors (2, 3, 4, 6), and the smallest to include as factors all four numbers (1 to 4) within the subitizing range. As a result of this increased factorability of the radix and its divisibility by a wide range of the most elemental numbers (whereas ten has only two non-trivial factors: 2 and 5, and not 3, 4, or 6), duodecimal representations fit more easily than decimal ones into many common patterns, as evidenced by the higher regularity observable in the duodecimal multiplication table. As a result, duodecimal has been described as the optimal number system.[4]"
https://en.wikipedia.org/wiki/Duodecimal http://dozenal.org/
Edit:
Another thing is that our current system of setting the duration of a second as a constant may be non-optimal for some purposes. Other systems let the length of time units vary with the seasons. It is interesting to think that under these systems, by maintaining the same base frequency, the tone of the note would change over the course of the year and with latitude. https://en.wikipedia.org/wiki/Roman_timekeeping