Optimizely has a somewhat distorted perception of "statistical relevance" in terms of test results. When we put Optimizely up to the test, we found that their platform was declaring a "winner" long before sufficient data had been collected. There's nothing specifically wrong with Optimizely, but it can, at times, paint an incomplete or otherwise misleading picture.

Regardless of what Optimizely's reporting dashboard tells you, you should always verify the data yourself to ensure that the result is accurate. http://www.evanmiller.org/how-not-to-run-an-ab-test.html

Optimizely is a really excellent tool and it provides an essential service to our company. Prior to Optimizely, we used an in-house a/b engine which required constant support by technical resources. With Optimizely, the company can make snap decisions and test them immediately.