Hacker News Reader

Comment by CookieCrisp | Hacker News Reader

I feel like that example is missing some context - if signups did increase then their experiment was successful - we aren’t here to make pretty pages, we’re here to make money.

View on HN

The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users. Most A/B tests focus on the former, very few on the latter. We had a free plan, and most users who signed up never made a single API request. So, assuming that the increase in signups is driving more business is just foolhardy.

You can always track signup/paying-users ratio. Purpose of landing/pricing page is to get the users to sign-up. Unless some dark pattern or misinformation is used to confuse users into sign-up, more users is a positive thing.

The problem with A/B test optimization is that, unless you're extremely careful, they naturally lead you to apply dark patterns and misinformation.

Ok, but the example we're discussing is one where the signup button was simply moved to a different position on the page. That's not a 'dark pattern'.

> We had a free plan, and most users who signed up never made a single API request.

That doesn't sound like a signup problem; what was the goal behind the free plan? Drive more paying users? Raise company profile? Lock in more users?

> The problem is that it's easy to prove that signups are increasing, and lot harder to prove that there was a measurable increase in number of paying users.

Okay? The A/B test sought to measure which of two options A and B led to more signups.

> So, assuming that the increase in signups is driving more business is just foolhardy.

Your "A/B test enthusiast" was not testing for or trying to prove a causal relationship between increased signups and more business.

If he made the claim separately, then that is the context that is missing from now multiple comments.

Reply on news.ycombinator.com