Comment by AJ007 | Hacker News Reader

Comment by AJ007 | Hacker News Reader

They increased the number of people submitting their URL, there is no telling if that actually resulted in higher leads for them.

What I have found is a simple landing page, that tells the user exactly what you are providing, and is free of any confusion, works the best over the long term.

I've run hundreds of thousands of website visitors through Google Website Optimizer in multi-variate tests and what I've found is that over time there is little to no difference in conversion rate for minor landing page changes. The biggest jumps come from eliminating content in the design and clarifying the message.

Looking at the small amount of users they sent to this landing page, I would call the results inconclusive. You can ramble off statistics to me all day long, but you can't change the fact that humans don't behave when the predictable that coin flips and physics do. (its really chilling when you see how many drugs the FDA has approved over tiny margins of change/success.)

> Looking at the small amount of users they sent to this landing page, I would call the results inconclusive

The statistic say otherwise. A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result.

> You can ramble off statistics to me all day long, but you can't change the fact that humans don't behave when the predictable that coin flips and physics do.

And you can rattle off personal anecdotes like this all day long, and the statistics are still more correct than you are and assert their margins of error and accuracy. The statistics are more correct than your intuition.

Keep in mind that the 1% margin of error applies to the measurements of the conversion rates. The 29% has a significantly higher margin of error.

> A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result.

Well, no, it's a set of numbers with percentage signs after them. Perhaps the documentation for Optimizely specifies how their error margins etc. are derived, but nothing in the linked article does as far as I can see. Without knowing that underlying reasoning, all those pretty graphs and percentages are just a load of gobbledegook, apart from the original data points and the percentage increase figures derived directly from dividing them.

What a lazy ignorant statement. If you want to see how they crunch the numbers, go look.

I did. A Google search for

"error bars" site:optimizely.com

turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written.

If you have found other sources where the Optimizely site publicly describes their statistical methodology, please share them. I think several people following this discussion would be interested.

Otherwise, I stand by my earlier comments.

So ask them; they aren't stupid, they wouldn't be building a business based on A/B testing without using valid methods of testing and displaying results. That you call the results crap because you don't have the perfect details of everything is simply absurd.

I didn't call the results "crap". I am simply pointing out that they are meaningless without knowing the methodology behind them. (And we aren't just missing the "perfect details of everything" here. As far as I can see, we have no rigorous details whatsoever.)

I would remind you that you were the person who was attacking another poster's position based on your interpretation of those currently meaningless numbers. It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.

You called them gobbledegook, same difference.

> I am simply pointing out that they are meaningless without knowing the methodology behind them.

Only if you assume incompetence or malice on the part of Optimizely, neither of which you have any valid reason to do. It's perfectly reasonable to assume they aren't stupid and the results are valid.

> It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.

Um, my claim is don't assume they're idiots; that doesn't require me to back anything up.

The poster I replied to wasn't attacking them, he was attacking statistic in general, which is what I was replying to.

Your response was to imply that Optimizely doesn't know what they're doing and therefore their results are invalid until you see how they're crunching the data; that's simply absurd.

Great point, the goal we talk about in this blog post is the number of people who click that button.

One nice feature of Optimizely is that we can test as many goals as we like-- you can add goals even after the experiment has started running and we retroactively measure the conversion rate!

After I read your comment I went in and added a goal to see whether there was a change in the number of people who ended up starting a free trial later down in the funnel after seeing each of these variations. Turns out there was a +2.5% increase for the "Give it a try" variation. :)

I'm curious how far you're able to follow them down the funnel (e.g. conversions from that free trial to paying member? or effect on the cancel rate of those members?). I ask because I've been looking at Hubspot, and find their ability to connect the lead gen into the CRM a big selling point (I guess they call this "lead nurturing"). Are you focusing narrowly on the lead gen half of it?

Hi Josh,

Yup, you can track as far down the funnel as you like. Here is an article in our knowledge base that explains how you can do this: http://support.optimizely.com/kb/goal-tracking-and-reporting...

Let me know what you think!

Josh -

I think tools like Optimizely can track to the final sale if you have an online (ecommerce only) transaction. This works well when the user is expected to complete the transaction online and without changing computers or deleting their cookies during the consideration process.

If you have an offline sales process (where a salesperson takes an order in person or on the phone) or customers who pay by invoice, or a longer and more considered sales process (bigger ticket items), you need a closed loop marketing tool that connects your marketign leads database to the CRM that the sales team is using. HubSpot does this, and connects to Salesforce.com, NetSuite, Highrise, Sugar CRM, Microsoft Dynamics, ACT, Goldmine, or pretty much any CRM.

So, Optimizely for small ticket / ecommerce, and HubSpot for large ticket / offline sales.

Thanks, Mike Volpe (CMO @ HubSpot)

Reply on news.ycombinator.com