Less is More: How SeeClickFix Improved Engagement By Reducing Visual Clutter

Dan from Optimizely here. I agree that engagement isn't always the best metric to use to determine the effectiveness of a variation and in this case it probably would have been better for them to track the submissions of the main form on their homepage. That said for large whole page changes it is a great way to determine whether or not people are more likely to engage on the page in some way vs. bouncing.

The original inspiration for Optimizely came out of my experience as the Director of Analytics for the Obama 2008 campaign where we used Google Website Optimizer and Omniture Test & Target. One of the things that was frustrating with GWO was only being able to measure one conversion goal. With Optimizely you can measure as many goals as you like and see the holistic impact of the changes you make. In this case I think they should have also measured the conversion rate of the main form and see the impact this variation had on it. We will do a better job of evangelizing lessons learned and best practices for our customers going forward!

Appreciate the feedback and keep it coming!

I imagine with their home page they could measure engagement best by keeping track of the percentage of people who enter their zip code to begin a report.