it's a tiny little more complicated than that.
The key element to grasp is that for all units above "second" aren't fixed: the duration of a minute, an hour, a day or a month or a year varies, depending on when it starts.
It's complicated by the fact that colloquial expectation - and likely often legally binding - is that "one day from now" is tomorrow at the same wall clock time.
So just expressing time deltas as seconds doesn't cut it, either.
This is on top of pesky details such as leap year rules (if multiple of 4 but not of hundred unless multiple of 400), half-hour time zones, and DST decided anually by parliament.
TAI and GPS - even if available - don't help you there because these things affect the algorithms and - worse - the data you need to store.
If you have a local time and forgot to store the UTC offset, you don't have a time. If you have a duration of one day, and haven't stored when this day starts, you don't know how many seconds in that day.
The particular problems of leap seconds are two: first, they are easily forgotten - leading to incorrect calculations. Second (heh!), since insertion of leap seconds is not regular, you need updates to keep your time arithmetics correct.
---
Now it might surprise you that I'd personally think leap seconds are cool, and we should keep them. However, it's hard to explain to someone faced with these problems why a few seconds of offset in sunrise time is such a big deal.