Yes-- sample size, and effect size. He's saying that the results were strong enough that given the sample size, they were statistically significant.

Simple example: if I have a study of 20,000 people and I find that 51% of people prefer foos to bars, that may be similarly significant to your study of 20 people, all of which prefer foos to bars.

If I've understood it correctly, the smaller your sample size, the bigger the difference you need between response rates. So if the difference is 60%, then a sample of 30 people can be statistically significant. But if you're looking for a 5% improvement on something, you'll need to sample a large number of people.

I just used this:

http://www.prconline.com/education/tools/statsignificance/in...

You are right.

The author should plug the numbers in here: http://www.usereffect.com/split-test-calculator

I'd go as far as to consider the data to be worthless without the confidence value