How Coderwall grew over 50% with an A/B test
rankscience.com
rankscience.com
1. How were the experiment groups chosen? You have to be really careful with SEO tests because some pages might get 100x the traffic of another page, so saying you split up 20,000 pages randomly isn't enough for a test like this. It is only meaningful if you split up 20,000 pages that had similar traffic profiles and were getting enough aggregate traffic to be able to notice an increase.
2. SEO tests take a long time so if you are doing it, I'd recommend more than 2 variations. You need to put the test up, wait for Google to index it, then wait a few weeks to see how the traffic changes. So since your turn-around time is at least a few weeks and maybe longer, try 4 or 8 variations if your traffic can support it.
3. I prefer my A/B tests to be a little more crazy, or at least try a crazy variation among more normal ones. In my experience the biggest gainers (and also the biggest losers) are ideas that seem crazy. Getting your feet wet with adding "(Example)" to the title is fine, but also try a crazy variation that is completely different and seeing how that does. Give yourself a chance to be surprised by your audience so you can learn more about them. And if the crazy variation loses by a lot, you have learned something important even if it isn't a winner.
4. The facts are a little dubious here. Traffic is up 50% over a timespan... there is no control. Traffic generally goes up for growing websites even if you don't do anything. You should also show your work on the 14.8% increase and give us some error bars. You say it is significant, how did you calculate this? It does seem like the test was better, but it is also important to make valid claims about it. Intellectual honesty when running A/B tests is really really important [1].
1: http://danbirken.com/ab/testing/2014/04/08/pitfalls-of-bias-...
Would love to chat more about your experience with a/b testing for top of funnel and hear more about your work. Can you email me if you'd like to meet up for coffee in SF? dillon@ranksci.com
Is this actually true? Shouldn't the fact you're choosing your pages from a very large set of pages negate this concern? Yes, you might have outliers, but your method of deriving significance should account for this.
Given that we don't know the specifics of the tests (this is a case study after all and we should expect details to be elided), I'm not sure it's fair to opine as you are on their quality. I would guess a company focused on this specific problem is both aware of and actively applying all the best practices you've mentioned.
For example, one of my sites (champsorchumps.us) has a landing page for every pro sports team. Here is a graph of the traffic to all of those landing pages in the past month I just pulled from GA: http://imgur.com/cB2igLc
Note: The content on each landing page is basically the same (of course with different data for different sports teams), but as you can see the traffic to each page is vastly different.
The top page gets nearly 25% of the traffic. Most of the pages get little or no traffic. If you consider the top page, traffic on that particular page goes up and down randomly all the time. If I have that page in a bucket with other pages for an SEO title test, and its traffic happens to randomly go up by 50%, the variance from that page alone might be equal to a naive "significant result" for the whole test.
You can still run a title A/B test (something I do on basically every site I run), but you have to be thoughtful about it. You have to consider the buckets carefully and consider what gains would be significant before you run it.
I'm not suggesting the authors of the post didn't think about it. Maybe they did. However I've talked to a lot of people about A/B testing and most of them don't. The problem is that all of these A/B testing posts always yada yada over the important parts of running A/B tests, proper setup and impartial analysis, and shoot straight to whatever variations they used and talk about the huge gains. So when people read them they think all they need to do is come up with some fun variations and boom, they are going to get huge gains. It just isn't true. If you aren't careful with A/B tests you can just as easily move backwards than forwards if you skip over the important parts of creating a test.
1. I would have randomly assigned pages with more than X average conversions to the versions, and then kept a count of how many pages saw an increase or a decrease.
2. The key phrase is, "If your traffic can support it." This is a question of doing statistical calculations in advance and deciding how big a random mistake you'd be willing to accept if you were unlucky. I find that people tend to err on the side of trying to get too much information out of too little data, so I would generally push towards fewer versions, not more.
3. I strongly disagree with you on this. I would prefer to run 5 small tests over one big one because you learn more. In particular I have found many times that titles, subject lines, and the like can have a surprising impact. The area of your site with the most likely big wins from A/B tests is where you have large numbers of unengaged visitors that you are trying to get to make the snap decision to click - which is exactly what this title test measures.
4. They are trying to run a business, not conduct a research product. Once they had a 14% known win on their hands, and enough sample data to be certain of it, there is no business value in continuing to do what you know is less effective. Do as they did. Collect overall figures, say "some fraction of that is our test", and don't try to measure exactly how big a fraction it is.
But to those points I would add a fifth. I have found that there is a significant "novelty" effect. For example if you're carrying on an email campaign, people will initially respond well to pretty much any subject line change the first time they see it. But not so much 3 weeks later. I doubt that was an issue here, but it is a problem to always keep in the back of your mind as a question.
Would be very curious how they calculated significance here.
S[it] = b0 + b1 * X[it] + b2 * A[i] + b3 * D[it] + e
S[it]: number of sessions originating on page [i] at time [t]
b0: base traffic for all pages
A[i]: base traffic for page [i]
X[it]: seasonal traffic for page [i] at time [t]
D[it]: dummy variable, 1 for after treatment 0 for control and before treatment
b3: effect size of treatment
e: noise/error term
E[S[i]] = b0 + b1 * E[X[i]] + b2 * A[i] + b3 * E[D[i]] + e
S[it] - E[S[i]] = (b0 - b0) + b1(X[it] - E[X[i]]) + b2(A[i] - A[i]) + b3(D[it] - E[D[i]]) + e
S[it] - E[S[i]] = b1(X[it] - E[X[i]]) + b3(D[it] - E[D[i]]) + e
(S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b1(X[1t] - E[X[1]] - X[0t] + E[X[0]]) + b3(D[1t] - E[D[1]] - D[0t] + E[D[0]]) + e
(S[1t] - E[S[1]]) - (S[0t] - E[S[0]]) = b3(D[1t] - E[D[1]]) + e
The key assumption is that X[it] - E[X[i]] = X[jt] - E[X[j]] for large numbers of pages. I found this assumption does hold in practice. I'm not a statistician so forgive me if I've made any statistical errors but I think this analysis is correct.
I might write up a blog post at some point to explain it in more detail.
Which one of these results would you click on? (query: mysql split string)
What exactly grew and how much is 50%? Are you getting 15 hits per month, up from 10?