Increase Signups by Adding Video to Your Page
abtests.com
abtests.com
It could just be that I am noticing the phrase since I read about it, but it would be a strange coincidence to see the exact wording so often.
I'm sure they don't think that a featured test would result in identical results on a different site. But opening up data like this for peer review and replication is a good thing.
Edit: Oh. dcancel said as much, just above this.
We at ABTests.com are simply trying to get more people to incorporate A/B Testing into their development process.
Thanks for the feedback, we appreciate it.
There were also other small subtle changes like the addition of "Request Instant Price Quote" headline etc.
A/B tests need to be as specific to one adjustment as possible.
I've noticed flaws like this many times on ABtests.com. But I guess they are like any war stories, its the overall idea that counts not the details.
What are the primary flaws you're seeing?
In your case adding a headline to the form was overlooked.
You were using form submissions as a metric from what I can tell. So changes specifically to the signup form itself (such as the header) is likely to have as much of an impact as the video.
Did you track how many people watched the video then signed up as a metric? That may be more telling.
If you tested by running one for 7 days and the other for 7 days, there are numerous other possible factors. Even the comparison of Friday versus Saturday traffic could have enough variation on a small sample to skew results. Internet congestion might have slowed the site down during one test, awareness might predispose people to be more likely to convert beyond trial or even the fact that the video version renders more quickly to the surfer proving that html design and page render time may have been the deciding factor. I didn't see any mention on the site how the trial was actually conducted, but, based on the other test results, I don't see enough isolation of variables to decidedly say the video was the sole factor.