- Use of analytics tends to replace user trials/interviews entirely, trading away rich signals for weaker ones
- Analytics can be used to justify otherwise unpopular or ill-advised changes
- When combined with certain changes (e.g. making features harder to access), the numbers can be “steered” in a particular direction to favor a particular outcome and better enable the last point (“Looks like nobody’s using that thing we hid behind an obscure feature flag! Guess we’re safe to remove it entirely now!”).
In theory telemetry/analytics have strong potential for improving software quality, but more often than not they’re just massaged and misused by product managers bent on pushing the software a particular direction.
Yeah, this is huge. The 30-day A/B test is a scourge on the industry.
Telemetry in the hands of stakeholders whose stakes are business/career KPMs will probably serve those, and the software experience will follow.
All this to say, I don't think Mozilla is doing much with all the telemetry data it's gathered all these years