Comment by jermaustin1 | Hacker News Reader

Comment by jermaustin1 | Hacker News Reader

Parent Full thread

I don't buy this article at all.

I am no longer with the company, but around 10 years ago, I turned a failing consultancy into an SEO SaaS company. Turning revenues from -40k/month to 90+k/month after laying off more than half the staff when we lost our last client and deciding to take our destiny into our own hands building a suite of SEO products.

We ran split tests on almost every design change. They weren't always tiny, some were entire page layouts. But even on things like changing the header text saw measurable differences.

I built the ability to split test into every facet of our framework. We had the ability to split test pages, transactional emails, marketing emails, and we would tweak everything all the time. If we noticed group a wasn't performing as well as b, we would drop a.

There were some tiny things that made major differences:

- Email "From" being a person's name and not just the service bumpped onboarding but nearly 20%.

- The onboarding subject "Almost Done..." performed about 10% better than "Activate your account"

- Having the video auto play converted visitors almost 50% more often than them having to click the play button (it may be evil, but it makes money).

- Having the video be an animation vs just a spokesperson talking into a webcam converted 5% better with the same script (+2% better with just reading the text from the page).

We had a large audience after we launched our first product free of charge for a limited time (linklicious.co -- formerly linklicious.me -- formerly lts.me), then used that momentum to built out a new product every month or so for the next couple of years. It was a great job in a terrible industry. 80% of my job was inventing new ideas.

-Take your email list -Split it in two equal halfs -Send each half the same email -Open and click rates will differ between the two lists

But if you apply statistics, you will see that the open rates only differ significantly 5% of the time, if you use a 95% confidence interval.

It will only differ 5% of the time if you have an adequate sample size and only check for significance once the sample size is reached.

If you end the test the moment the data reaches 95% significance, it will show a difference about 50% of the time for the same email. Many people make this mistake.

A 95% confidence interval doesn't mean much if you dont follow good statistical practices.

I don't understand why you would need a sample of a certain size. Setting a significance threshold at 5% takes sample size in to account. For example, if I ran a permutation test with a sample size of 5 in each group it could never been significant at that threshold, and never is < 5%!

A small sample size would lower your power to detect meaningful differences, which the original scenario doesn't have (by definition).

(If distributional assumptions, etc, are violated, then that's a different story!)

You need to make sure you have enough samples in order to know if you rejected the null hypothesis by chance. Stopping your test early, is a form of p-hacking. See:

https://heapanalytics.com/blog/data-stories/dont-stop-your-a...

Peeking at your data, and calculating the sample size you need for a test are separate statistical issues. I agree that peeking messes up significance levels :).

The point I was trying to make was you can decide to run a test with a very small sample (e.g. n = 5), and it will still have the level of type 1 power you set if you chose a significance level of .05.

> You need to make sure you have enough samples in order to know if you rejected the null hypothesis by chance.

You do this when you decide the significance level (e.g. .05). The value needed to reject, given a significance level, is a function of sample size.

The definition of Type 1 error on wikipedia has a good explanation of this:

https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

It should be possible to adjust the significance estimate down based on an assumption of continuous sampling less than the sampling size target. E.g. if you're aiming for a sample size of 100, and after 20 samples the split is 19/1 the current ratio is 95%, but that's not accurate yet, so e.g. it could be adjusted to the average between the two possible outcome extremes (99/1 and 19/81) and show 59% with low confidence. I don't know the statistics or if that specific method actually makes sense, but it should be possible.

Only bigcos or companies whose core business is DTC will have the resources to do proper ab tests and then they might not have the knowledge to make them statistically significant.

What you're saying is perhaps correct but it's not how AB testing is done in the real world. I think many tests show a difference for random reasons.

a) 5% is still one in 20. Are you doing 20 trials or more a month? (Of course, it won't be _exactly_ 5% of the time... there's some other statistics we could calculate to say how likely it is to differ by more than some specified delta to 5% depending on how many trials you run... oh my)

b) That's assuming you are using statistics right. It's quite hard to use statistics right.

That's way better odds than most people's gut feel. Everyone thinks they don't need to run experiments because THEY already know what works.

Did you know, in experimentation programs run by Microsoft, Google and Amazon, roughly two thirds of their ideas have no impact or hurt the metrics they were designed to improve? And yet rookie web Devs or marketing assistants "know" better.

Source: https://www.google.com.au/url?sa=t&source=web&rct=j&url=http... (Read section three of this paper by a Microsoft distinguished engineer)

It's actually more like 80% for Google. I spent a lot of time running various experiments on Search.

I'll point out a major difference, though: Microsoft, Google, and Amazon are already highly optimized. They've had millions of man-hours put into optimizing everything from product design to UI to UI element positioning to wording to colors to fonts. When you get a new, beneficial result in a change to Google, it's usually because the environment has changed while you weren't looking, and the user is different.

That doesn't apply to a new business that's just learning how to sell their product. In a new business, by definition, you've spent zero time micro-optimizing your message & product. You can get really big wins almost by accident, if you happen to have stumbled into a product category where users actually want what you're building.

Assuming you completely randomize which half goes into each split, it wont vary significantly over time.

Obviously some segments are going to open/click/spam at higher rates, but that's fine and controllable with a customer rating. So when you split your users, ensure an even makeup of customers, and your tests will be fine.

Also known as an A/A test.

> Having the video auto play converted visitors almost 50% more often than them having to click the play button (it may be evil, but it makes money).

Because the ones who weren't going to convert never came back

It may or may not be true that they never came back. A good split test tool will tell you that as well.

I'm increasingly of the opinion that split-testing tools are the hidden way Satan influences the world. If your A/B tests keep telling you that you should make the product more user-hostile and less useful, that you should manipulate the users and abuse the commons for short-term gains, something is very wrong - with the tools, the tests, or the world.

Split testing can be used by companies who care about their users and companies who don't. It just tells you how your users react. It's not a substitute for user empathy or regulations.

How many impressions were you running if you were able to measure a 2% difference?

If you want to be completely sure, a really small amount, lets say your baseline is 20% to detect a 2% difference you just need 2 x 6,347 views for each test

Check this tools[0] and this article [1]

[0] https://www.evanmiller.org/ab-testing/sample-size.html#!20;8...

[1] https://www.evanmiller.org/how-not-to-run-an-ab-test.html

When we implemented split-testing-by-default baked into the software itself, we saw maybe 1000 visitors a week from various sources. Our free conversion rate was at about 3% and once we were "fully optimized" we were closer to 10-15%, and our paid conversions were about 10-30% of the free users depending on the month (December and January always sucked).

Most of these numbers were measured after at least a week of leaving the experiments running.

We never tested more than experiment at a time per visitor, but our framework had the ability to run multiple tests simultaneously, while cookie-ing a visitor to a single test.

Reply on news.ycombinator.com