Do you use Google Optimizer, something else, or have you coded your own?
If you're not currently using A/B tests on your site, why? If you are using A/B testing on your site has it been useful?
The designer created 5 variations of the merchant landing page. The variations had different color schemes, placement of coupons, etc. We used GWO to test the original versus the new variations, using an exit through an affiliate link as our goal. In this round, the original beat out all of the new designs (60%+ conversions vs 30%).
Back to the drawing board. What was it about the new design that was affecting conversions? An informal focus group revealed that the listing of coupons looked too much like display ads, and people didn't like clicking on ads. The designer came up with new designs that looked more like a blog. We re-ran the tests with 5 variations, but the original still beat out the new versions though by a smaller margin (60%+ vs 50%+). The focus groups liked the new design better than the original, but real world users weren't clicking as much.
That's when I stumbled on the answer. Whenever we did focus groups we showed users the design in a conference room, and it was always on a local server. Whenever our QA department did testing, it was always from our office, 1 mile from our datacenter. When I sent a friend a link, he thought the site was too slow. I suspected that since the new page was slower than the old one, people were giving up before it fully loaded.
We re-ran the GWO tests, this time with various stripped-down versions of the landing page with different loading times. We even created one version that loaded even slower. Finally we had a winner: when the new design was stripped down a little, and the images heavily optimized, and aggressively cached, we got the load time down to about the same as the old version. This was the version that beat out the original (60%+ conversions vs 72%). This is the version on the live site today.
Thanks for the insight.