Hacker News Reader

Comment by gnaritas | Hacker News Reader

> Looking at the small amount of users they sent to this landing page, I would call the results inconclusive

The statistic say otherwise. A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result.

> You can ramble off statistics to me all day long, but you can't change the fact that humans don't behave when the predictable that coin flips and physics do.

And you can rattle off personal anecdotes like this all day long, and the statistics are still more correct than you are and assert their margins of error and accuracy. The statistics are more correct than your intuition.

View on HN

> A 29% bump with a 1% margin of error is not inconclusive; it's virtually the very definition of a conclusive result.

Well, no, it's a set of numbers with percentage signs after them. Perhaps the documentation for Optimizely specifies how their error margins etc. are derived, but nothing in the linked article does as far as I can see. Without knowing that underlying reasoning, all those pretty graphs and percentages are just a load of gobbledegook, apart from the original data points and the percentage increase figures derived directly from dividing them.

What a lazy ignorant statement. If you want to see how they crunch the numbers, go look.

I did. A Google search for

"error bars" site:optimizely.com

turns up exactly three hits. One of them is the blog post we're talking about. The others are discussions on the Optimizely support pages from December 2010 and January 2011, which are similarly statistically waffly. The older one promises a further clarifying post that never seems to have been written.

If you have found other sources where the Optimizely site publicly describes their statistical methodology, please share them. I think several people following this discussion would be interested.

Otherwise, I stand by my earlier comments.

So ask them; they aren't stupid, they wouldn't be building a business based on A/B testing without using valid methods of testing and displaying results. That you call the results crap because you don't have the perfect details of everything is simply absurd.

I didn't call the results "crap". I am simply pointing out that they are meaningless without knowing the methodology behind them. (And we aren't just missing the "perfect details of everything" here. As far as I can see, we have no rigorous details whatsoever.)

I would remind you that you were the person who was attacking another poster's position based on your interpretation of those currently meaningless numbers. It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.

You called them gobbledegook, same difference.

> I am simply pointing out that they are meaningless without knowing the methodology behind them.

Only if you assume incompetence or malice on the part of Optimizely, neither of which you have any valid reason to do. It's perfectly reasonable to assume they aren't stupid and the results are valid.

> It's up to you to back up your claim, not up to the rest of us to figure out whether your argument has any merit.

Um, my claim is don't assume they're idiots; that doesn't require me to back anything up.

The poster I replied to wasn't attacking them, he was attacking statistic in general, which is what I was replying to.

Your response was to imply that Optimizely doesn't know what they're doing and therefore their results are invalid until you see how they're crunching the data; that's simply absurd.

Keep in mind that the 1% margin of error applies to the measurements of the conversion rates. The 29% has a significantly higher margin of error.

Reply on news.ycombinator.com