Ask HN: What is the best way to do visit based A/B Tests?
While I am familiar with testing typical page-to-page conversion, at this project we are looking at differences in users' behavior in a total visit. For example we are looking whether a change in 'people also bought'-algorithm which is shown at multiple places on the site is changing total conversions on a per visit level. So we look op total visits for a segment, then total conversions and then look at differences between A and B in conversions/visit.
To get data for the two versions we create segments in Google Analytics. The problem I currently have tough comes partly from sampling issues. - Since we have millions of pageviews per day, even on a daily level I still get a sample rate showing only 15% of visits - Because the conversion is relatively small per visit (~0.1 per visit) the data sampling issue seems to get even worse - Regardless of the data sampling issues, I think we should have per visit based data in order to be able to calculate the variance of the particular conversion of interest occurring with which we can than compare the means and test whether it's significant.
To show the problem I did some experimenting with using our methodology and splitting our control group in two and then test as if the one half of the control group was a B version. Using our current, flawed, methodology I get 'significantly' different results for both groups.
So I have a couple of questions: - How do you guys handle visits based AB tests? (and is this even a thing?) - Are there any tools that capture per visit data like described above? - Do you know any good resources on AB testing that go further than simple page-to-page tests but really get into testing at a visit level?
Any thoughts are greatly appreciated!