Hacker News Reader

This is something about A/B tests that always bothers me. They always seem to assume that if variation A is best for a proxy stat (signups, views, etc) it's also best for what I'm really interested in (longterm value). I still use them though - not sure there's much you can do about it anyways.

View on HN

On the other hand, if you can eventually measure long-term value and you can keep track of users over a long enough cycle, then you can eventually run an A/B test on long term value. This depends on accurately tracking users, having a quantifiable way to calculate long-term value, and eliminating other sources of error, but it should be absolutely doable.

You can also measure how well your proxy stat does correspond to long-term value and adjust your expectations accordingly.

Reply on news.ycombinator.com