Thanks for the info.
Question though, how do you get these measurements? Out of X people who visit the website who don't log on, how many will sign up vs not?
Question though, how do you get these measurements? Out of X people who visit the website who don't log on, how many will sign up vs not?
The brief version is that you cookie each visitor with a random unique identifier. Each identifier maps to one of the versions under testing in a durable fashion. The first time you see an identifier for a particular test, you increment the participant count for that appropriate version. When a conversion -- here a signup -- happens, you look at the identifier, check what version they saw, and increment the conversion counter. From then it is just a math problem. So when you see a conversion rate like 24%, that implies that 24% of people who viewed a page the test was active for signed up prior to becoming lost to the system. (By, for example, leaving forever, clearing cookies, etc.)
Although one could theoretically exclude folks who log in later from the participant count, I think that is a poor use of your time for most people. Existing site users will be split across all alternatives evenly, so their failure to sign up for the site affects all alternatives easily. Since A/B testing doesn't really care about the exact value of the conversion rates and focuses on the differences between them, that comes out in the wash. (Plus, for many services, first time visitors swamp existing users of the service, so even if you were worried about distortion it would be minimal.)
The metric we used in the a/b tests was signups and the participants are anyone who lands on the homepage (new or returning).