It matters for some things. Without those fictional leap seconds, the sun would be 29 seconds out of position at local noon, for instance.
It most certainly matters to a lot of people. It sounds like you've never met those people.
This is ignoring the fact that due equation of time, solar noon naturally shifts around tens of minutes over the course of the year.
To drive the point, for example local mean solar time at Buckingham palace is already more than 30 seconds off from Greenwich time.
There's very good reasons that are important behind why we try and keep UTC near UT1, so saying "it doesn't matter to anyone" without even entertaining that some people might care isn't very constructive.
Generally, it's useful for midnight to be at night, and midday during the day. UT1 is not regular, so you need some form of correction. Then the debate is about how big and how often.
It is up to you to keep TAI for everything and let your representations of physical coordinates drift away into the galaxy or something, but that's not the majority choice. Overwhelming majority choose UTC time.
TAI is still nice for many high precision applications, weirdly including a lot of precisely those geo-spatial use cases, so we have both.