How two magical words increased signups by 28%
visualwebsiteoptimizer.com
visualwebsiteoptimizer.com
I'm going to try things like, e.g., Start Making Bingo Cards.
I actually had a few dozen buttons made in a particular design to test different calls to action against each other earlier this year, but the design itself flopped -- hugely -- against my pre-existing buttons, so much so that I didn't think color or copy changes on it would recover the loss. Curses on having my testing bandwidth limited by graphical expertise...
I'm running a test right now, with "try it for free" currently at 262 out of 792 clicking through, and "get started now" at 269/778 (so up 4.5%). It's not yet conclusive, but at no point in the test has "try it for free" outpaced "get started now".
If my results ever show themselves to be "conclusive", I'll post about them at abtests.com. I'd encourage you to try "get started now", though, to see what you find.
Actually I disagree. The phrase "sign up for free" contains some hint of a potential "but", similar to "free to join" or "download trial for free", or other carefully worded phrases that litter anything-but-free internet offers.
"it's free" is far more emphatic and open.
Having said that, a hint of a potential "but" (i.e. trial-ware) is good at times so that the users don't take the free part for granted and they are aware of an upgrade option.
Effectiveness of this kind of test should not just be done at the spot where you make the change but end-to-end.
It's very well possible that by placing the 'it's free!' there the signups shift to a much larger number of people that will never convert to paying members (for a freemium site).
If you only test at the point of making the change it is very well possible to actually increase free sign-ups but to decrease your conversion rate to paying members.
So, this test is about an intermediary goal, the real conversion is the rate between new visitors and paying members.
I sure hope that your overall conversion rate to paying members shows a similar increase, but make sure that really is the case.
>To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers
We measure 3 related goals in GA, the homepage -> signup form; homepage -> signed up user; and new visit -> paying user (using the goals but also e-commerce values).
Looking back and learning from this exercise, in the next tests that we're doing we'll be adding different UTM campaigns to the different variations (in VWO) and be able to track conversions to paying customers further down the track. I cannot stress enough how much valuable information this gives.
In any case we CAN see the increase in homepage -> signup form: http://cl.ly/503bf3943cc390e4f715
This did increase the actual signup conversion as well (homepage -> signed up user): http://cl.ly/e6403e90307c58c8a6e7
The tricky thing is to drill down and see which users these are and where are they coming from and how are they using Soocial. Any tips are welcome! ;-)
In order of highest conversion rates are: New Zealand (1.16%), United States (0.98%), Brazil, Australia, Ireland , UK (all around (0.79%), Singapore, South Africa (around 0.6%) then it levels out.
Which is totally rad :)
Now if only you could modify the video on your site or post a blog post to explain... how does my site get loaded into your 'visual editor?' Does all my css stay there? etc. The video makes it look like the site has magically been loaded into the editor and everything's intact :P
Go, try the live demo http://dev.visualwebsiteoptimizer.com/first_time_ab.php?demo...
Do you think we should blog about the fact that you don't need to do any thing special to load a site into our system?
After seeing this link, it's totally clear to me and i will definitely sign up when we're ready.
I'm trying to do a test, where one variation has this text box and nothing else (kind of like Google homepage). My only worry is that it may spook some less technically savvy people into thinking we have hacked their website!
I'm not completely sure I'm correct (please correct me if I'm not!) here but as I understand it the article does not support the claim in the headline. The headline claims that signups increases by 28% in the changed version and that this was all attributable to the change.
It's the second bit that isn't supported, they say that the result was statistical significant but what I understand them as saying was that it was statistically significant that the new variation was better than the control. But it could be better by an amount more or less than 28%, all we know from that is it's almost certainly (95%) at least a little better. We would need to know the number of trails to be able to get a certainty for the amount of improvement.
Could someone with a slightly better understanding of statistics chip in maybe? I could use some more information in my own A/B tests, sometimes I know a change is going to be a pain to maintain so I want to know not just if it is better, but by how much.
If you have any specific question, please feel free to ask.
How are you using to arrive at that conclusion though? When I do a A/B testing I (or rather, the software I use) use the chi-squared test to give a confidence value expressed as a percentage. When I get over 95% I know I have statistical significance and I end the test. At that point I also have an "improved by" number but I would have to collect much more data for that to be statistically significant as far as I understand.
However ...
I believe it is flawed methodolgy to run a test until you get a significant result. I'm pretty sure I read something lunk to from HN that discussed this at some length. It's possible - if you simply run your test until you get significance and then stop early - that you will get significance because of random fluctuations in the middle of your trial and stop early when you shouldn't.
As I recall, you should decide on the length of your trial at the beginning, then run your stats at the end.
I'll try to find the article in question, but my Google-Fu is pretty poor today for some reason.
EDIT: It's here:
Seems my understanding was even less than I thought :)
When a car's top speed increases from 105mph to 120mph, after you modified the engine, you say it increased 15mph ... or, perhaps you could say it increased 14.3%.
Using this paradigm, when a conversion rate increases from 14.5% to 18.6%, you would say it increased by 4.1% ... or, perhaps you could say it increased 28.28%
Woops! We have a problem. We just expressed the same increase, in the same units, using two very different numbers.
This is obviously a problem. The reader doesn't know what you're talking about.
It's perfectly valid to say the car increased by 15mph, so it must be perfectly valid to say the conversion rate increased by 4.1%. In fact, I think that when most people read something like this, this is what they assume, that the delta between the old conversion and the new conversion is that percentage. At first blush, saying the conversion rate increased by 28.28% should be OK, but it's not OK, because the whole point is to convey information, and you have failed to do so. Now you have to use context, to figure out what the hell they meant. This is unacceptable, especially when talking about mathematics, we should not need context to sort out what's going on.
What's the answer? I don't know. But there's definitely a problem here.
No you cannot say it increased by 4.1% and that's because you are talking about per cent. I don't know why you would ever use 4.1% in this context. It is simply wrong.
May be you can say 4.1 percentage points but I don't see a need for it when you can say (and be right) that it increased by 28%.
To be sure, the fact that we are talking about percent changes something, which is what I spent several paragraphs exploring.
To say that subtraction of percentages is against the rules, (while perfectly valid and often used in other units) but percentages of percentages is not against the rules, requires a bit more than "It is simply wrong", don't you think?
Sure, you're "right", when you say it increased by 28%. The point is, perhaps you shouldn't be "right", and what is "right", anyway, if nobody knows what the hell you're really saying?
We see this when talking about tax changes. If somebody wants to minimize the sticker shock of a tax increase, they use the delta. So, if the rate rises from 3% to 4.5%, they say they increased the rate 1.5%. If you want to maximize the sticker shock, you say that the tax rate was increased 50%. Either side can say they are doing the math "right".
Those who care about actually conveying the information, rather than produce sticker shock of either sort, always have to spell it out carefully, by actually saying, "the tax rose from a rate of 3% to a rate of 4.5%" This is a bit longer, but it's necessary, given this weird unit.
Edit: It doesn't work.
That can't be good for conversions.