Hacker News Reader

Comment by MattBearman | Hacker News Reader

Speaking as someone who turned their 5 year side project into a full time startup, I whole heartedly disagree with point 1 - split testing is likely to be a waste of your time.

Unless you have a tonne of traffic, and let's face it, side projects tend not to, A/B testing simply doesn't work. I've tried it a few times with BugMuncher, but the results take 3 - 4months, and are usually inconclusive. I've spoken to other people in similar situations and they've found the same thing. For reference, BugMuncher is lucky to see 3,000 uniques a month :)

I believe A/B testing is a good idea when you have the traffic for it, and I'd love to be able to make use of it, but unless your side project is getting 10s of thousands of unique views each month, there's much better things to spend your limited time on.

View on HN

Wholeheartedly agree here. As a one-man-band on most of my side projects, I simply don't have the time or resources to create multiple A/B cycles, with iteration after iteration until I think I've got the best conversion rate. In any case, as it is a side project and not my day job or primary skills, I could be A/B testing until the cows come home, and still not hit upon a magic landing page that improves my conversions.

A/B testing is best left to larger scale projects where you can dedicate a proper resource to it.

I came to the same conclusion after running small-changes A/B test.

But I think it still makes sense for smaller projects to make "big" A/B test, like testing a completely different homepage/landing page with different copy and marketing approach.

Another example of useful A/B test on small scale: for registered users, testing if an automated "support" email is helping moving customers to the next step of your funnel (A: send email B: don't send email. Measure the conversion of that specific funnel step)

Yes, agreed. Ab testing is great, but without sufficient traffic it's really counterproductive.

I'd also add that being casual about test statistics in the way this post describes is only going to mislead you. If you can't take the time to understand the math and run proper experiments, you're better off not testing at all since you'll just end up cherrypicking statistical outliers over and over while thinking you're doing something useful.

I agree with this, you need traffic to measure it, 3000 unique visits a month is right there at the cusp of what is needed to see results. It took an average of 1000 uniques per split to see meaningful results. Also unlike scientific journals a negative result IS a result. If your conversions didn't change, that means you change didn't affect your audience's decision to buy (not great but also could have been MUCH worse ;) ).

Agreed. The problem with split testing on low traffic is that you can't be sure what's causing the difference. It could be the time, location, device, etc. You need to split into cohorts of very similar users and compare their response, and you just can't do that unless you have high traffic.

What is your startup?

BugMuncher - https://www.bugmuncher.com I've been transparently blogging about my experiences going full time and trying to get profitable without outside funding - it's been a pretty crazy year but I'm still on target :)

I read you blog post tilte "My Timeline of Failed Business Ideas" [0] It was a interesting reading, I can relate to that. I think everyone here have a similar timeline after a few years in the business.

[0]https://www.bugmuncher.com/blog/timeline-of-failures/

Reply on news.ycombinator.com