There were 12 (24) clicks total, you don't even have to use p-value or any other calculation to know that this result means pretty much nothing ...
The reason it feels intuitively broken is because the conversion rate is so low. But there were about 10k impressions in his test.
Can't be 95% confident in something if natural variability drowns out the effect. That kind of conversion rate is one bot RNG away from returning to the mean (noise), it just takes some domain knowledge to recognize why p-hacking is an illusion.
It's not uncommon that people fail to notice the nuance here and it's quite important for correctly interpreting reults.
So there's about 3 standard deviations between the two, this sounds like quite a bit but really means they're both 1.5 standard deviations from the supposed mean. Which is, not great, though it might pass some of the weaker statistical tests.
Now you should actually weight the values by the total number of impressions in which case you might get a slightly higher significance since the one with 12 clicks was seen by more people.
So on the balance you should be wondering what you're paying the graphic designer for, but perhaps not start a new career designing low-budget adds.
12 vs 24 clicks is not significant, it could’ve gone either way. Also given this minuscule sample, it’s easy to conduct p-hacking to get the desired outcome
In other words, if I'm doing the math right, for him to p-hack this by rerunning the experiment in the case of no difference, he'd have to run it more than 100 times to get a 50% chance of getting as good or better significance.
There's of course plenty of other things that could be wrong outside of the simple statistical test, he could be making the numbers up, the groups might not be properly randomized etc.
Linkedin calculates an ad impression (last I checked) as 50% of the ad is on screen for at least 300ms. I can be scrolling as fast as my thumb will flick down my LinkedIn newsfeed and it would probably count.
Then the KPI being used is clicks. I don't know of any business owner who would take that as valid. It should be some kind of conversion event (newsletter signup, contact request, or purchase, etc.)
If it were up to me, I'd want 4,400 clicks and a few dozen conversion events to do my calculations on statistically significant effectiveness. Especially since the author is paying CPC (cost-per-click)... who cares about impressions at all?
It's a statistically significant difference.
That seems like a highly significant result to me...
The fact that it has a significant p-value is interesting, but the lack of information about how the author decided when to stop is suggestive of p-hacking (i.e. we don't know how many screenshots were taken, but we understand that the author posted only the most favourable one)
R> prop.test(c(12, 24), c(5563, 4403))
2-sample test for equality of proportions with continuity correction
data: c(12, 24) out of c(5563, 4403)
X-squared = 6.5211, df = 1, p-value = 0.01066
alternative hypothesis: two.sided
95 percent confidence interval:
-0.0059903650 -0.0005970741
sample estimates:
prop 1 prop 2
0.002157109 0.005450829But in this case I suspect people were just curious why the ad looked like that and clicked to find out. Those may not be the people likely to convert.
Especially if the second option is cheaper and faster, there's IMO no bayesian prior that the professional ad being better (the null hypothesis) is true.
So... seems like useful data to me.
With a prior that A is superior, you may still end up believing that A > B after updating, because there’s just so little data.
And my main point is that a 60% probability is in fact actionable in the real world, in a situation where you are forced to take action with incomplete information. Assuming you are running an ad campaign, you have to choose one of the two.
P=.95 still is an arbitrary threshold, even if it's a commonly used one.