What's the use of real-time results for A/B testing? Most likely, it's only going to lead to erroneous conclusions, as discussed in this thread - http://news.ycombinator.com/item?id=1778104.
Here is an excerpt from an email discussion I had recently that touched on http://www.evanmiller.org/how-not-to-run-an-ab-test.html.
Evan Miller has a point, but not as good of one as he thinks.
It is true that multiple peeks mean that eventually any test will find significance at any level you want. However in A/B tests the peeks are not independent. This greatly weakens the effect he is talking about.
Section 7 of my presentation, starting at http://elem.com/~btilly/effective-ab-testing/#slide59, is about the question of how long it takes for a test to complete. For that I ran numerical experiments with constant peeking, literally every time you add one to A and one to B you peek again. You can see graphs of how many errors there were, and how long it takes to get an answer.
Here are key points:
- Be suspicious of tests that end quickly. Run them a bit longer on general principal. (In general I'd call 500 people a very small test.)
- Nobody can predict how long a test will take. Even if you know the actual improvement, you still can't predict time to within an order of magnitude.
- If a test has been running for a long time you know the true difference is small, so there is no harm in accepting whatever answer it gives.
During the Obama campaign we were frustrated by having to wait 24 - 48 hours to get results for our experiments. We were lucky to be getting a ton of traffic and hence be able to get statistically significant results in just a few days.
Another big benefit of real-time results is helping our customers answer the question "is this working correctly?" Seeing the data come in in real-time helps reassure folks that the tool is working properly and collecting data.
Whoaa why isn't there a crazy big testimonial screaming that on the homepage?
[EDIT: Sorry I misunderstood as you having used optimizely for the campaign, but it sounds like you probably thought of the idea as a result of your frustration.]
I can understand customers getting frustrated for having to wait. But, that's probably what will make them jump to conclusions. I don't know how much you want to get into advocating "best practices", but it would probably be good to ask the user to set their minimum number of visitors beforehand.
The "is this working correctly?" question sounds like a very good practical issue you picked up during your beta testing.