Wipers and climate are automatic in Teslas. If someone is messing with the mirrors while driving, then I think the five whys lead to careless driving, not a bad UX.
It's a valid criticism that their execution hasn't completely achieved the goal. But if you think the goal is wrong, then yeah, that's a great reason to avoid Teslas, because they're going to keep removing more and more controls and readouts as they automate them or make them irrelevant.
It's the people who don't know to do this who merge into you because their mirrors aren't adjusted properly making you invisible to them.
This is impractical in cars that don't save driver profiles.