> I just don't think this is a universal problem with the methodology
In an isolated world I would agree with you, but in the messy reality we live in I think the broader problems with the methodology are two:
1. The belief that everything can be measured. There are many intangibles (user trust, willingness to put up with bugs, "vibes") that aren't easily measurable. Yes, net promoter scores etc, but every company I can think of that uses them builds bland, buggy, largely disliked products that people use only because they have no choice.
2. Focusing on experiments and measurable outcomes creates a tendency to de-prioritise anything that isn't easily measurable. For example larger, riskier projects that can't be quickly tested. Or whimsy - easter eggs that devs added to many products in the past that users remember for years. Rarely added anymore because they can't be justified against a roadmap full of experiments.
Sometimes it feels a bit like we're sitting around so focused on measuring whether guests prefer one dish or the other and arguing over which experiment is best, that we don't notice a forest fire is blazing outside.
To me maybe it's a bit of a question of science vs art. Take the video games industry for example, the AAA studios are largely moving toward the science end of the spectrum with predictable games that extract maximum engagement, while indie studios are mostly building small, unique experiences that don't try to dominate your attention.
Experimentation always narrows focus down, and I think it's worth considering what's being traded off by doing so.