The hidden costs of A/B testing
stickermule.com
stickermule.com
The article is right that some ideas aren't worth testing. For example if your checkout page is broken, just fix it.
One mistaken idea a lot of people have is that half your A/B tests will win and you'll get instant magic results. This myth comes from multiple places. Vendors and CRO agencies push case studies about wins, not about the process (which includes lots of losses). Until last year all A/B testing tools (including the one I work for) were nothing but false positive engines [1] due to support for multiple goals without applying Bonferroni, using non-sequential tests, and similar issues. Even if you just ran a bunch of A/A tests you'd get lots of wins!
[1] This is still true to some extent. The situation is now better - VWO, Optimizely and A/B Tasty all have fairly solid stats. There are a lot more than these 3 vendors, however.
(Disclaimer: I work for an A/B testing vendor.)
https://blog.optimizely.com/2010/11/29/how-obama-raised-60-m...
I'm a big fan of running A-B tests on major product changes - which is basically just standard business benchmarking. Measuring the impact and understanding the results of a change are important whether you run a web app or a flower shop.
PPC is the one place for me where smaller AB tests are absolutely essential. In most other scenarios there are too many confounding variables to really understand much from small tweaks in a product.
Like ANY other tool, A/B tests have their place, but the incremental gains you're likely to produce from a different color button pale in comparison to what else you might be able to do with that mental energy.
Also completely agreed that paid traffic is a completely different beast.
Are they? Of course everyone thinks their ideas are "obviously good", but testing often proves otherwise. That's the whole point of it.
Using your example of "obviously good", lets say something seems "obviously good", so you launch it, and after a week, you see that it converts worse or better. That's an A/B test, where A is before the change and B is what is after. You don't need software for that, you don't need to do on-the-fly A/B testing, you can just launch.
Given a site has existed for a while, it has a pretty known A for most things you'd want to test. Doing A/B tests on small elements seems may indeed not be worth the extraneous costs of time etc.
Exceptional growth doesn't happen because you're great at a/b testing. It happens when your team is good at decision making period.
It sounds as though the author needs to combine the "How we improve conversion without A/B testing" section with user tests and data analysis to build stronger hypotheses. Since we've put more effort into the research phase, our success rate has improved dramatically.
The real value of A/B testing is validating meaningful hypotheses that help you learn what matters to your customers, not unrelated individual improvements. By only observing the overall trend, you miss out on this.
I wrote about this recently here: https://medium.com/@nickboyce/5-steps-to-a-better-a-b-testin...
The point of the article isn't to say a/b testing doesn't work or shoudn't be done. It's simply to say many companies are mis-using their resources by following the standard thinking on a/b testing.
We could spend another month figuring out how to get better at a/b testing or we could spend our time on another activity that produces a higher return on time spent. It turns out that's what we did and the return has been much better.
The author doesn't mention what kind of tests were performed, but often when I read about the "futility" of A/B testing, it's usually due to lack of up front preparation and discussion of the tests true objectives. So the classic "red button vs. green button" might get you a result in the testing software, but it doesn't necessarily translate to more sales/leads or whatever the ultimate goal is.
A solid a/b testing process doesn't just happen without time & effort. You need someone smart analyzing user behavior and forming hypotheses as well as a talented design team to develop concepts that test them.
We simply acknowledged the costs of doing a/b testing well and decided our resources are better spent elsewhere.
A few notes:
>I should admit that we still A/B test some ideas. Rather than adhering to a strict A/B testing process, we let our team to A/B test ideas they're curious about.
The idea is that you don't test everything in the first place, only test things where your thinking is "My hypothesis is that changing or adding feature X will improve certain metrics" -- and in those cases, you a/b test. Else you aren't really proving that you're right or wrong. If it doesn't matter if you're right or wrong, why implement feature X at all? If it's a qualitative improvement, it doesn't make sense to a/b test it.
>3) Performance >Related to lost conversions, A/B testing tools make your site slower and this also reduces your conversion rate--even for the control. The additional conversions you lose because of this performance hit are another, often ignored, cost of A/B testing.
You should probably be using a different tool if it makes your site slower.
>5) Speed >A/B testing slows down your organization's decision making. Some ideas are obviously good and adhering to a strict A/B testing process reduces your time to go live. Time is finite and the number of improvements you implement per year has a major impact on your growth trajectory.
It's impressive that growth is simply a factor of time within this company. This makes sense only if their average production decision making has a positive impact on growth/conversion rates as a factor of time.
>You can monitor your long term conversion trend rather easily via Google Analytics to gauge if you're making good decisions.
Yes, but that would be very unactionable. If the growth is going up, you pat yourself on the back and say "I'm clever" and if it isn't you say "I suck". But since you don't know which decisions or what the real reason is for conversion going up or down, you don't know what the fundamental metrics are that drive your growth up or down. You basically say "I don't care why things are going well" or even lie to yourself and say "Things are going well because I make good decisions."
Growth is not just about growing, but about knowing why you are growing and using that knowledge to find new ways to grow. If you know the fundamentals about what drives your growth, you also know which things you can or cannot change without affecting growth.
Any recommendations? I see a lot of tools out there that work by basically rewriting the page on the client side, which will tend to give you suboptimal performance. And if your app's already a JS heavy SPA thing, your over optimistic byte budget is probably already running a deficit.
To take another contrasting opinion. Who cares if you know exactly why you're going. We know in a general sense why we're growing: we constantly improve the experience of our service and make customers happier.
It's quite hard to maintain aggressive year over year growth targets. If you're pulling it off it really doesn't matter if you figured out why via a/b testing or not.
Obviously, we'd prefer stronger YOY growth without knowing the exact causes than weaker YOY growth because we wasted resources on a/b testing to get precise answers to why we're succeeding.
Beyond that, I wasn't referring to trending growth via GA to see if you're making good decisions. You can trend conversion rate by channel to see if you're getting better over time. Obviously, if conversion is going up and to the right for most channels you're making good decisions.
The '6 hidden costs' can all be addressed, some are not even hidden like time, running tests take time and resources like anything else. The other factors like speed/performance and confusion can all be solved very easily.
Some of my take-aways for testing
- Don't focus on small changes like buttons, copy etc.. these are mostly useless yes
- Make radical changes, e.g. on stickermule can try to have the whole product to make a sticker to checkout on 1 page or start with configuration first and then select sizes etc..
- To filter out false positives, compare same sample of data (e.g. organic traffic is good) for the same period range before a winning test went live and after
Personally I have seen some great results with the above tips in mind, from B2B making visitors perform small actions first or showing a demo vs. presenting them with a form immediately (4x more leads) to e-commerce sites adding a direct checkout vs. normal checkout path (lower avg. order but much higher conversion rate to largely make up for it) etc..
You could argue these are part of product (which a conversion department should be part of, not marketing) but without doing the testing you can only guess about the outcome. And sure many tests are fixing usability (e.g. seen 250% increase in registrations by fixing telephone number prefix and date of birth field formatting) and UI issues but that's part of it.
The biggest point being #1: Resources are finite. Time spent on A/B testing cannot be spent elsewhere. How you use your time determines your growth trajectory. A/B testing is a growth tactic, but it's not always the best use of your resources.
There is absolutely no way to solve this problem. You'll always have a trade off to make when it comes to how you spend your resources and time. You shouldn't neglect that cost when deciding if a/b testing is necessary for your organization.
Btw, keep in mind, I admit we still do some a/b testing. It's just not an key component of our growth strategy.
What I mean is that don't see it anywhere as a hidden cost in the true sense of that word as you know it takes resources and time, but more so about wrong priorities and business value.
As for the other points, they are so easy to solve:
>>Lost conversions
Trade off you need to make beforehand and need to be aware of with the end goal in mind which is to improve your main conversion metrics.
>>Performance
Most decent conversion platforms use fallbacks and also your way of setting up experiments makes a big difference.
>>Confusion
Letting other departments know what you're doing and what's running, might be harder in big corps though.
>>Speed
This goes for everything in a company I think (new feature/product releases, quick fix vs. proper solution etc.). I personally don't mind spending x amount of time to improve y % in conversion rate, so comes down to setting good KPIs and testing what really is important to your business.
>>Focus
Can be done with a proper conversion team that has all these roles covered (bit hard to set up but possible) or making the conversion department a client within the company. Also the teams mentioned work for other departments as well and improving conversion is a win across the board not just for the conversion team.