Non-stationary A/B tests

The point is that if your metrics are not stationary, then the variance in observations will seem higher than they really are if you don't account for time. This means you have to collect a lot more data before being able to draw a statistically strong conclusion. They describe a way of adding a time based stratification to the experiment data that doesn't have the weaknesses of simpler methods, like stratifying using a fixed interval size (like day)

The paper's intro mentions some of the issues. The biggest one IMO is a Simpson's Paradox issue that arises when (1) the difference between A and B is nonstationary, (2) the probability of being assigned to treatment or control is random at any given instant, but varies over time, being manipulated by dynamic optimization algorithms attempting to maximize signal / profit / etc.

Aside - this paper isn't the first time this issue has been noticed, or even the first time it's been addressed with rigorous statistics. Optimizely did it back in 2017-2018: https://support.optimizely.com/hc/en-us/articles/53262137051...

It would be nice if these authors mentioned the prior art and explained how their work improves upon it.

Consider a brick and mortar shop doing an A/B test on which poster brings more people into the shop: one of an ice cream, or one of a roaring fireplace.

It doesn't matter how randomly you divide your A and B groups, if you run your test in summer, you'll get different results to running it in winter.