A few notes:
>I should admit that we still A/B test some ideas. Rather than adhering to a strict A/B testing process, we let our team to A/B test ideas they're curious about.
The idea is that you don't test everything in the first place, only test things where your thinking is "My hypothesis is that changing or adding feature X will improve certain metrics" -- and in those cases, you a/b test. Else you aren't really proving that you're right or wrong. If it doesn't matter if you're right or wrong, why implement feature X at all? If it's a qualitative improvement, it doesn't make sense to a/b test it.
>3) Performance >Related to lost conversions, A/B testing tools make your site slower and this also reduces your conversion rate--even for the control. The additional conversions you lose because of this performance hit are another, often ignored, cost of A/B testing.
You should probably be using a different tool if it makes your site slower.
>5) Speed >A/B testing slows down your organization's decision making. Some ideas are obviously good and adhering to a strict A/B testing process reduces your time to go live. Time is finite and the number of improvements you implement per year has a major impact on your growth trajectory.
It's impressive that growth is simply a factor of time within this company. This makes sense only if their average production decision making has a positive impact on growth/conversion rates as a factor of time.
>You can monitor your long term conversion trend rather easily via Google Analytics to gauge if you're making good decisions.
Yes, but that would be very unactionable. If the growth is going up, you pat yourself on the back and say "I'm clever" and if it isn't you say "I suck". But since you don't know which decisions or what the real reason is for conversion going up or down, you don't know what the fundamental metrics are that drive your growth up or down. You basically say "I don't care why things are going well" or even lie to yourself and say "Things are going well because I make good decisions."
Growth is not just about growing, but about knowing why you are growing and using that knowledge to find new ways to grow. If you know the fundamentals about what drives your growth, you also know which things you can or cannot change without affecting growth.