How many users do you need to start split testing? | Hacker News Reader

How many users do you need to start split testing?

5 pointslionesscg7 comments

We recently launched this website, wannity.com, and are having pretty much visits than we expected (around 1k/month). We have low conversion rates right now, and we're wondering if it's time to start split testing, or the number of users is not enough to start making decisions after these tests?

Thanks.

View on HN

Right now, instead of split testing, do usability testing. Get 5 to 10 people who are close to the kind of people you expect to be visiting your website (or buying from you), and ask them open-ended questions.

Believe me, at the stage you're in, you'll learn far more than whatever split-testing you can do right now.

Here are a few questions I could think of (in no order):

1) On a previous visit to the website, you saw a bookshelf that was in the form of a bent arrow. What would you do if you wanted to buy it now?

2) If you wanted to keep up with all the new products on Wannity, can you find a way to do that?

3) Why would you not share a product you like in Wannity on your social networks?

4) What could be made better about the Checkout process?

5) Why would you not buy from Wannity?

Also, one small issue: I can't see the complete form in 1366 x 768 (http://puu.sh/8nkKK.png). If most of your buyers/visitors are on larger heights, you're good. Else maybe a bit of restructuring is in order.

Thank you so much siddharthdeswal. We made a Google form in order to get qualitative data, but this will be much more productive. Thanks for the bug reporting also.

If you do many small tests, you can pick confidence level less that 95%. Even if you call tests at 70% confidence, as long as you are iterating rapidly, you'll end up ahead in the long run. Also, you can revisit the same tests later. As long as your testing ideas aren't prohibitively expensive in time and implementation, there's very little risk at calling tests below 95% confidence.

Surely you can split-test at any level, it just takes longer for useful data to build up?