I think there's an aspect of the recently-discussed right-hand pole instability: an A/B test continues just long enough to see an increase in a given metric, but basically never long enough to determine whether that increase results in a change in behaviour that results in a longer-term net-decrease in that metric.

e.g.: Putting emojis in your notification texts increases engagement… until it kills your engagement entirely.