Any time that is supposed to represent both a wall-clock time has to deal with it.
There is currently no known answer to the question "What is the interval between the unix time stamps @1720026000 and @182002600" However, there is a well defined answer to "What is the UTC time for @182002600" and thus also "what is the local time in a timezone with UTC offset X for @182002600"
If we were to redefine Unix time to use Atomic Time, then we could answer the first question but not the second.
> We add an entire day to the calendar about every 4 years. It's not a problem because everyone is aware of it. So I think the only real thing that needs to change about leap seconds is awareness of them.
It's not just "about every 4 years" it's precisely "every 4 years, except for centuries, except for quad-centuries." If you give me any year, I can tell you if it will be a leap year. I can program a non-internet connected device and it will get leap years right indefinitely. Leap seconds are not predictable, but rather determined empirically and announced about 6 months ahead of time. Any device that wants to properly account for them needs to be updated at least every 6 months to be correct.