Redesigns are often self-indulgent. Designers like that they get to do something new, employees who stare at the same software every day get to change things up, and managers get a highly visible change they can point to as evidence of their "impact". What's best for the users is often not a top concern.
Most people who don't really care about tech that much don't like UI changes like you said because it means relearning what they know
The ones who love flashy new UIs are the tech enthusiast, the ones who love tech but use them on a surface level only, they are also the ones who will buy new, unproven tech and care little about privacy issues or open source. My guess is it makes a lot of sense for big tech groups to target them instead of the grey beards who won't be convinced anyway, right as they may be.
I have rarely seen UI changes where users were genuinely excited to have a new UI with the understanding that they'd have to learn new paradigms. Most web apps should still be Bootstrap apps, but of course then you can't put that on a giant dashboard wall at a conference ;)
like I would hope the users love it, to justify the work of a new design?
It’s going to be an uphill climb for users to adjust to a new UI. Broadly speaking, that’s fine. If the payoff is worth it. What’s the payoff?