Much more important, though, is how it affects the future. The fact that timestamps in the past might be a few seconds different from the straightforward “now minus N seconds” calculation is mostly a curiosity. The fact that clocks might all have to shift by one more second at time point in the future is more significant. There are plenty of real-world scenarios where that needs some substantial effort to account for.
It most certainly matters to a lot of people. It sounds like you've never met those people.
This is ignoring the fact that due equation of time, solar noon naturally shifts around tens of minutes over the course of the year.
To drive the point, for example local mean solar time at Buckingham palace is already more than 30 seconds off from Greenwich time.
There's very good reasons that are important behind why we try and keep UTC near UT1, so saying "it doesn't matter to anyone" without even entertaining that some people might care isn't very constructive.
Generally, it's useful for midnight to be at night, and midday during the day. UT1 is not regular, so you need some form of correction. Then the debate is about how big and how often.
It is up to you to keep TAI for everything and let your representations of physical coordinates drift away into the galaxy or something, but that's not the majority choice. Overwhelming majority choose UTC time.
TAI is still nice for many high precision applications, weirdly including a lot of precisely those geo-spatial use cases, so we have both.