Comment by jacquesm | Hacker News Reader

Comment by jacquesm | Hacker News Reader

Parent Full thread

hey Spif,

Effectiveness of this kind of test should not just be done at the spot where you make the change but end-to-end.

It's very well possible that by placing the 'it's free!' there the signups shift to a much larger number of people that will never convert to paying members (for a freemium site).

If you only test at the point of making the change it is very well possible to actually increase free sign-ups but to decrease your conversion rate to paying members.

So, this test is about an intermediary goal, the real conversion is the rate between new visitors and paying members.

I sure hope that your overall conversion rate to paying members shows a similar increase, but make sure that really is the case.

Actually, spif mentions the following in the post:

>To be honest we aren’t sure yet [of what impact this makes to the actual signups]. We wonder how these tests measure up to the goal funnels in Google Analytics and compared to actual conversions into paying customers

Ah, cool, I missed that. Thanks. It really is the crux though, any kind of partial measurement is essentially meaningless without the overall picture.

This went live before we launched our premium account so don't have those stats to track.

We measure 3 related goals in GA, the homepage -> signup form; homepage -> signed up user; and new visit -> paying user (using the goals but also e-commerce values).

Looking back and learning from this exercise, in the next tests that we're doing we'll be adding different UTM campaigns to the different variations (in VWO) and be able to track conversions to paying customers further down the track. I cannot stress enough how much valuable information this gives.

In any case we CAN see the increase in homepage -> signup form: http://cl.ly/503bf3943cc390e4f715

This did increase the actual signup conversion as well (homepage -> signed up user): http://cl.ly/e6403e90307c58c8a6e7

Fantastic info. Should have included in the case study ;)

Only just looked up those numbers as per Jacques question. ;-)

Thank you, that puts it much better in to perspective.

Reply on news.ycombinator.com