In Frequentist thinking; p=0.05 means that if there was in reality no difference in your A and B and you repeated the experiment many times, 5% of the observed differences would be equal to or greater than the difference you just measured.
No probabilistic statement about the results being correct or incorrect can be made from a Null-Hypothesis significance test.
If you have a few minutes to spare, I would very much welcome your thoughts so that I can either correct the article, or take it down - The last thing I want is for it to sit out there on the open internet as misinformation.
My goal was to create a framework which — while less mathematically accurate (hence “rhetorical device”) — helped convey the seriousness of making business decisions based on P = 0.05 to people for whom 95% statistical significance doesn’t mean anything. And clearly, based on reactions here, I failed at that goal.
So, if you’re game, I’ll quickly to walk you through my thinking, and you can help me understand where I went wrong. Best way to contact?
This is correct and the original post is wrong.
The p-value (in my understanding) makes a prediction about what would occur if the experiment was repeated infinity times.