The statement I see on that page is "There is a 91% chance that Variation A has a higher conversion rate".
I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.
I am not sure how to interpret that. We would have to dive into the GitHub repo and figure out which test it performs I guess.
So, assuming a uniform prior and updating with 39/3000 and 28/3000 conversions the difference between the two distributions is greater than zero 91% of the time. It's only guaranteed to be above zero at about the 80% credible interval, and since we started with an uninformed prior that'd be about p=.2?
I'm open to correction here.