Well, it's hard to change the number of solar days it takes to go around the sun one full revolution. Anything different and we'd see the seasons shift.

As for number of hours in a day. 24 is very divisible which is better than say 10. You can divide it into 1, 2, 3, 4, 6, 8, 12, 24 without fractions.

As for 60 minutes and 60 seconds, these again are very divisible: 1, 2, 3, 4, 5, 6, 10, 12, 15, 20, 30, and 60.

The more interesting question, should we have chosen base 12 instead of base 10. It would of course be harder to count using your fingers.