Hacker News Reader

He seems to assume that the contribution and volunteer rate of the added marginal people (those who signed up after seeing the best performing media / button, but would not have signed up with the default media / button) is as good as the rest of the people (those who would have signed up regardless).

View on HN

He doesn't mention it, but that's easy to determine. Simply see what the average contribution of people who contributed during the A/B testing was per image/button combination. If there not all about the same then there may be some effect (negative or positive) that the added marginal people brought. If they're about the same, I think its a decent assumption to believe that it will be the same going forward.

And of course, you can simply validate it by seeing if it is the same going forward.

Simply see what the average contribution of people who contributed during the A/B testing was per image/button combination

We didn't measure the average contribution for each variation for this particular experiment but in future splash page tests we measured eventual $ per pageview for each variation and it was very closely correlated to % signup rate.

Reply on news.ycombinator.com