Hacker News Reader

Yea, but what are they using to analyze your results? You literally have no idea how robust (or not) their statistical methods are. It's been my experience that 3rd party solutions are notoriously shoddy in this department. They have no incentive to be robust.

View on HN

AFAIK (it's been over a year), they use intent to treat approach, and only remove users from the experiment after they have already been matched to an ad. This allows them to ensure that the users are matched in terms of expected conversion rate, which allows for valid results.

I believe that there were rumours of using bayesian analysis in the future, but I believe it's a two-sided p-value based on the differences in your chosen outcome.

Reasonably simple, and mostly effective.

Reply on news.ycombinator.com