Hacker News Reader

Comment by iamleppert | Hacker News Reader

A/B testing is high on hype and promise but low on actual results if you follow it though to actual metrics. I've done various forms of A/B throughout most of my career and found them to be cinsistent with the OP's results.

A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.

That, or actually build a product that users want instead of chasing after pointless metrics. I mean, really, you think changing the color of text or a call-out is going to make up for huge deficiencies in your product or make people buy it? The entire premise seems illogical and just doesn't work. The only time I've seen a/b tests truly help was when it accidentally fixed some cross browser issue or moved a button within reach of a user.

Most of the A/B website optimization industry is an elaborate scam, put off on people who don't know any better and are looking for a magic bullet.

View on HN

How is this the top comment? It's recommending changing your website without knowing what the effect is, which is terrible advice.

I've personally run tests and seen tests achieve statistically significant (and very large) results that weren't merely fixing bugs or moving buttons within reach of a user. It could be things like adding a modal, changing copy (yes, literally just adding or changing words on the page), removing content, changing an image, or reordering funnel steps.

Think about that: by merely changing some sentences, your website could be 10% more successful next month. But you won't know whether it's working without a/b testing.

You don't just create tests randomly, but based off of an understanding of the product, goals, signup funnel, etc. which may involve talking to users, but ultimately verifying the change is for the better through a/b testing.

I would agree that there are a number of people running a/b tests that don't understand the math or know what they are doing, but that isn't an indictment of a/b testing.

> I've personally run tests and seen tests achieve statistically significant (and very large) results that weren't merely fixing bugs or moving buttons within reach of a user. It could be things like adding a modal, changing copy (yes, literally just adding or changing words on the page), removing content, changing an image, or reordering funnel steps.

You should read the article. It explained very well why a) statistical significance is meaningless when split testing websites (you are guaranteed to archive significance if you run the test long enough; the math behind significance testing doesn't work like assumed by Optimizely et.al.) and b) most tools on the market are inherently unable to prove effect size by the way tests are designed.

Just do this short experiment yourself: Do an a/b test with identical versions of you site. Spoiler: With 100% certainty, you'll see both significant results and a sizable effect size difference.

    statistical significance is meaningless when split
    testing websites (you are guaranteed to archive
    significance if you run the test long enough

Wait, what? Not if you use the right test to evaluate your change. This article isn't showing that A/B testing is bunk, it's showing that Optimizely is using an inappropriate test.

    Do an a/b test with identical versions of you site.
    Spoiler: With 100% certainty, you'll see both significant
    results and a sizable effect size difference.

The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical. Running your A vs A experiment and evaluating it correctly you'll find results significant at the p < N rate N% of the time. And the longer you run it the lower your estimate of the effect size will be.

I think we agree. I should have elaborated more.

> Wait, what? Not if you use the right test to evaluate your change.

Sure, but how often to people actually choose the right combination of test, testing tool and estimated effect size for their significance to have actual meaning?

The problem with experiments like your typical website split test is the potentially infinite population. With a big enough N, the probability of reaching significance approaches 1.

> The whole point of significance testing is that, done properly, it tells you how likely you would be to get these results by chance if the two sides of the test were actually identical.

Yes, I know. But the magic lies in that "done properly". Have you ever approached a tool in the field that would fail my proposed test? I haven't.

    With a big enough N, the probability of reaching
    significance approaches 1.

Well, not with an A vs A experiment. Then the probability of significance at the p < 0.05 level should stay 5%.

On the other hand, in an A vs B experiment where B is very slightly different from A you're right that with a large enough population you become very likely to reach a p < 0.05 significance level, just with an extremely tiny effect size. The experiment has told you something about the difference between A and B, and you now have high certainty, but it's just such a small difference that you probably don't care.

    Have you ever approached a tool in the field that
    would fail my proposed test?

Your proposed test being, run an A vs A experiment and a tool is faulty if it concludes there's a difference significant at the 0.05 level more than 5% of the time? All the tools I've used for A/B testing would do fine on this test. This includes a custom tool at a computational advertising startup I worked at, Google's internal experiment platform, and Google Analytics' Content Experiments.

(If I didn't trust a tool to pass this test I'd just use it to get the counts for both sides of the experiment and do that stats on my own.)

A/B (or champion/challenger as we called it in the 90s) testing is fine. The problem with these tools is that they do not alleviate the need to understand statistics.

A common place where people get stuck when A/B testing is in a local minimum or maximum. For example, instead of tweaking button colors someone should instead redesign the whole form. Talking to users can help with this, but users are notoriously bad at not knowing what they really want.

The most common place people get stuck in a/b testing is proper design of experiments, i.e. absolutely no methodology (i.e. proper blocking) at all. It's ridiculous how a/b testing is perpetuated to be of paramount importance. A/B testing would be one of the last things I would do, if I worked at a small ecommerce company, for example. What lead to the current state is venture-backed companies and "growth hackers" and "conversion optimization experts" selling false promises to people who haven't read a marketing textbook in their life.

I'm very surprised this is the top comment too, where's patio11 and his ridiculously large amount of positive and useful split testing examples when you need him?

Anyway, on to your comment:

> A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.

I don't know what this means. Can you elaborate? What exactly is "significant instrumentation"?

> That, or actually build a product that users want instead of chasing after pointless metrics.

Pointless metrics? I'm not sure what metrics you measure but visitor engagement and visitor to trial conversion rates are the lifeblood of a software sales and marketing funnel.

When somebody in marketing considers A/B testing as a means to increasing trial and paying customer conversions, they're operating on the basis that their product already provides value. The A/B tests are a means of establishing the best method for communicating that value. Better communication = more trials. More trials = a larger pool of potential customers to convert to paying customers. It's pretty basic.

> Most of the A/B website optimization industry is an elaborate scam

What?? Had you said a number of people in marketing could be in danger of blindly relying on their tools (e.g. Optimizely) and not applying common sense or checking the maths I would have wholeheartedly agreed. In my personal experience with the various websites I market, and given the hundreds of split testing examples I've read from people I do not feel are in on the "scam", I've got to strongly disagree with this point.

Disclaimer : I work for an A/B testing platform.

There is nothing inherently wrong with A/B testing, and it's basically the only way to verify that anything you do has a positive impact. What's created a hype bubble is articles and platforms saying that minor UI changes, colour of buttons etc. make a real difference to conversion rates. Good tests based on well thought out hypotheses relevant to your actual customers, can and sometimes do help websites find measurable gains. It is not a silver bullet, and no matter how careful you are there is always statistical uncertainty, it won't always help.

>>A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.

In my experience this only works for obvious, glaring flaws, such as "I hate that the phone number field enforces a specific format" or "Why the hell are your password requirements so stringent?"

But a lot of UX is about what happens on the unconscious level. A user will never tell you, "Your bright red background feels too aggressive/hostile" because they simply don't recognize it at the conscious level to be able to verbalize it. The signup or checkout form might make them feel uneasy, but they won't know why.

I find that the best way to optimize websites is to do A/B testing first, and then talk to your users about the subset of the results (which might be the majority) that you find confusing or surprising. Basically, never rely on just one method, because then you will be hamstrung by the shortcomings of that method.

One underused but valuable use of A/B testing is to safely roll out new features--even if you aren't really interested in optimization. I've seen many cases where A/B testing has caught bugs that would have cost the company a significant amount of money and customer trust.

The overall utility of A/B tests is a complex topic and largely depends on the skill of the experimenter. Well-designed experiments are extraordinarily useful. A/B tests are one tool for product development, but they can't replace it.

I wouldn't call that A/B testing. It looks pretty similar under the hood but you'd normally call that a phased or gradual rollout and have different goals and metrics for it.

As you said, what cle described is a gradual rollout. You can do both at the same time though -- simply start your test on X% of the population and then roll it out to more people as you run your test. This is also a great way to run long-term holdouts.

A/B Testing is a way to conduct an experiment. Instrumentation and talking to users is another good way to gain insights, but it not an experiment. They are two different (and often complementary) activities.

Many, many people have successfully used A/B Testing. I've personally used it to great effect several times. I certainly don't make decisions purely based on the statistical results, but I find it to be an extremely useful input to the decision making process. All models are flawed; some are useful.

Perhaps an A/B test on A/B testing vs. not is in order.

At small scales it's probably better to spend your time working on the core product than to make small tweaks, but once you're operating at a large scale a 0.1% conversion bump from a small change (like making the cents display in the price smaller) can mean millions of dollars in increased conversion.

There is a big difference between conducting an experiment and understanding the results.

Even talking to users would not help much if one continues to build a faster horse.

Each page could contain a small feedback area in which the actual product user can specify what feature he/she would like to see improved.

Changing color of text is not much improvement, unless the text was unreadable or same color as background.

No amount of data driven can replace what the actual product user says.

The problem is that actual product users would probably say they want faster horses.

> A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.

But but but telemetry is evil!

Telemetry in a specific product? Fine.

Telemetry in my OS* that has access to everything? No.

* That I cannot turn off.

What if the OS is the product?

An OS is a product of course, but to me it's on a completely different level to "just" an application product. It has access to everything, I want to be able to trust it and know exactly what it's doing. Telemetry I can't turn off ruins that. Having it is fine, just let me turn it off...

I like Windows and really like VS, I was often literally the only (non-VM) Windows user in a sea of OS X at several offices but Windows 10, the flip-flopping UI, the ads, and the telemetry pushed me to Linux and building my own desktops.

A modern OS is nothing but a bunch of apps and the kernel. Most of the apps run in sandboxes.

Exactly. This is why Win10 is so bad. You simply cannot deactivate the telemetry at all (as an end user)!

Instrumentation can be activated with customer consent, if support is requested by the customer.

You described something with small benefits but limited downside, especially if your company has a team of a dozen or so people.

Reply on news.ycombinator.com