Assuming you’d want to work data-driven, what would be a better approach to figure out what works and doesn’t? Or is it a conscious decision to capitalize on the short-term at the expense of the long-term?
e.g.: Putting emojis in your notification texts increases engagement… until it kills your engagement entirely.
No amount of short-term testing will discover [*the cause of] a long-term decline, if the long term response of the system of customer/product/company is swamped by an opposing reaction in the short term.
The population is already changing long term for reasons outside of our control. If for instance, other companies are adding the emojis and people are tired of emojis that will affect the performance of our experiments.
It is by no means perfect, but at least it offers a way to get feedback (given enough users, not annoyed enough :)
The only real way to deal with this is to run these types of A/B tests “forever”, which is not desired. Which is why I’m very much wondering, what is the appropriate way to deal with this?