siddharthdeswal

I'd like to clarify a couple points here. The terminals have been in use for years now across petrol pumps, restaurants, malls, pubs etc. And the majority are the wireless kinds now where they bring the machine to you to punch in the pin number.

And the cultural part about the upper class finding it a walk of shame is, quite frankly, ridiculous. Instead, we feel secure giving out the pin number because we know that the person needs the card and the pin to make a transaction and we'll get an SMS the moment a transaction goes through. And more often than not, we're too lazy to walk to the terminal so we just tell the waiter the pin number.

But by the author's own admission, aren't people keeping the tab open so that they can come back to it later?

And is someone who consumes your content in 3 minutes more relevant/valuable than someone who is inactive on the page for 1 hour, but still spends 3 or 4 minutes of active time?

Interesting post. So how does this help someone make more money off their media/publishing website? What actionable insights can someone draw from inactive time?

Yup, that's the exact implementation am using, but just with JSON instead of an API. Do you have a link for your project?

I'd also ask if there are parallels to caviar, or most other luxury goods. Though caviar might have a demand-supply reason for being so expensive.

What did you tweak? I'm not a technical person so don't understand why those specific use-cases are bad. If you want the majority of Linkedin users to care about the cause, you ought to put it in language that the vast, non-technical majority will understand.

Don't A/B test your credibility. Not everything should be (or can be) subjected to hypothesis testing.

I think you mean Google Website Optimizer, not Visual Website Optimizer.

Visual Website Optimizer just introduced a $9 plan (I work there). I agree with your point in general, but the truth is most SaaS tools make far more profit when serving bigger customers.

Right now, instead of split testing, do usability testing. Get 5 to 10 people who are close to the kind of people you expect to be visiting your website (or buying from you), and ask them open-ended questions.

Believe me, at the stage you're in, you'll learn far more than whatever split-testing you can do right now.

Here are a few questions I could think of (in no order):

1) On a previous visit to the website, you saw a bookshelf that was in the form of a bent arrow. What would you do if you wanted to buy it now?

2) If you wanted to keep up with all the new products on Wannity, can you find a way to do that?

3) Why would you not share a product you like in Wannity on your social networks?

4) What could be made better about the Checkout process?

5) Why would you not buy from Wannity?

Also, one small issue: I can't see the complete form in 1366 x 768 (http://puu.sh/8nkKK.png). If most of your buyers/visitors are on larger heights, you're good. Else maybe a bit of restructuring is in order.

Companies 'might' have confidentiality agreements, but I think it would be naive to expect that in today's world news won't leak. So maybe they take that into account while going about such acquisitions?

And getting the 'first scoop' is the job of the fourth estate, so can't fault them for doing their job.

Thanks for the feedback. Tells us we need to think up better headlines.

Sample size was 4500+ distributed between the two versions. In this case, the sample size was not small but the conversion rate of the page was very low.

Which is why a 600% increase was even possible. That's usually a problem with A/B testing in the real world. There are so many websites whose conversion rates hover in the 0.xx% range. Thoughtful hypotheses almost always result in significant gains. Not because they did something magical, but because they simply communicated their business offer better.

They didn't measure how this affected revenue. They're still in the process of rolling out that test. (I work for VWO)

Trying to do exactly that at http://visualwebsiteoptimizer.com/ab-split-test-duration/ and http://visualwebsiteoptimizer.com/ab-split-significance-calc... but yeah, we need a better way to present this.

The tool has that built-in. As soon as significance is achieved, we show a "winner", so to answer your question, instead talking about statistical significance we focus on the more positive aspect of an A/B test.

Those not initiated in Statistics find this a little difficult to understand. I personally go about it like this:

Me: "If you flip a coin 10 times and it shows heads 7 times, can you say with certainty that the coin is loaded?" Them: "Umm, no not really"

Me: "Ok, what if I flip it 100 times and it shows head 70 times?" Them: "Yeah maybe"

Me : "What if 7000 out of 10,000?" Them: "Of course it's loaded"

And that's how I explain the basic concept of statistical significance, but without the math. More data means it becomes more 'obvious'.

To take it a notch further, I ask "If your life depended on it, would you still bet that the coin is loaded?" People usually become unsure after this and I joke that their previous decision was based on 95% significance, but if they really want to be sure, they should wait for 99% significance.

Exactly. For someone who's just starting with A/B testing, this serves as a good/interesting introduction. But for someone who really wants to understand the detailed behind-the-scenes, this will obviously fall short.

Coming to your suggestion, do you have a good book/website in mind?

Multiple species of fish are usually called "fishes". We used this explanation to verify http://grammarist.com/usage/fish-fishes/

Not sure if it is correct though. Do you have a link to a better explanation?

Those are good points, but we were gunning for the marketer who isn't familiar with A/B testing. This because we see that a lot of our potential customers have usually heard the term somewhere but need a basic, easy-to-understand explanation.

Do you think it achieves that purpose?

That's a beautiful landing page. Assuming you work there, how much time did it take to build?

Great work Kushagra. Very useful stuff.

True, we don't. But that is usually the reason for A/A tests throwing up different results (assuming of course, that the tool used to run the test is not flawed).

There's a very simple reason for that.

You're seeing a difference between Control and Variation in an A/A test is because a very small number of visitors have been tested. To explain, suppose you toss a coin 10 times and 7 out of those it shows heads. Just based on these 10 tosses, would you conclude that the coin is loaded? Probably not. Suppose you tossed the coin a 100 times, it'll probably show heads maybe 43 or 47 or 51 or 52 times.

Point being, as you toss it more and more, the number of times it shows a heads or tails comes closer and closer to 50% but you need to toss it a large number of times to be fairly certain that it isn't loaded. The more you toss it, the more certain you are. However, you'll only be more and more certain, but never completely certain. VWO works on a similar principle. The more number of times you toss up Control and Variation to visitors, the more certain you become of either being better, worse or equal to each other.

If you'll read the post, the graph shows the fluctuations in the beginning, after which things kind of settle down. In an A/A test, they'll settle down to a very similar conversion rate.

Here's an article from the VWO Knowledgebase that'll help you with running an A/B test correctly http://visualwebsiteoptimizer.com/knowledge/how-to-ideally-r...

Content that loads dynamically won't force you to use Multivariate Testing. In fact, most commercial services implement both A/B and Multivariate tests using client side JS.

Also, you can A/B test dynamically loading content so I think you just need to find out how to do it using your chosen software.

Nah, not missing it. Well aware of it actually but that was not reported to us by Veeam, so instead of trying to fudge, we just stick to reporting for now.

VWO rounds off 99.9x% to 100%. Sample size was 3500 visitors approximately to control and variation each.