In theory, the solution in situations like this is to continue the test over a longer time. Typically with ‘holdbacks’ – subsets of users who don’t get a feature for a long time. This is easier if you have an app that everyone uses because with a website, it’s harder to reliably find holdbacks who are also a representative sample (eg it won’t work so well to hold back everyone using the site in a certain language as those people may be statistically different from the general population in other ways).

There are still a bunch of problems – higher maintenance burden, harder to iterate on a site quickly. Though I think you identify what I would consider the bigger problem which is that they cause political difficulties as a holdback can only really turn around and say that the positive impact people claimed wasn’t really borne out in the long term. So even at places that do holdbacks, the results may be silenced or ignored. If a holdback shows something continuing to work, that’s hard to get excitement about even though I think one should expect many of these a/b test results to not have long lasting effects.