Hacker News Reader

Comment by shubhamjain | Hacker News Reader

The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Most A/B tests focus on the former, very few on the latter. We had a free plan, and most users who signed up never made a single API request. So, assuming that the increase in signups is driving more business is just foolhardy.

View on HN

> We had a free plan, and most users who signed up never made a single API request.

That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?

> The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users.

Okay? The A/B test sought to measure which of two options A and B led to more signups.

> So, assuming that the increase in signups is driving more business is just foolhardy.

Your "A/B test enthusiast" was not testing for or trying to prove a causal relationship between increased signups and more business.

If he made the claim separately, then that is the context that is missing from now multiple comments.

You can always track signup/paying-users ratio. Purpose of landing/pricing page is to get the users to sign-up. Unless some dark pattern or misinformation is used to confuse users into sign-up, more users is a positive thing.

The problem with A/B test optimization is that, unless you're extremely careful, they naturally lead you to apply dark patterns and misinformation.

Ok, but the example we're discussing is one where the signup button was simply moved to a different position on the page. That's not a 'dark pattern'.

Reply on news.ycombinator.com