>At 99% certainty, you have 99 people saying “yes” and 1 person saying “no.”
The article was worth the read to me just for this part. I thought I had a pretty decent intuitive understanding of probabilities but this put things in new perspective for me.
The rest of the article is marketing gobbledygook to me, though.
Let me quote from "0 And 1 Are Not Probabilities"[1].
> In probabilities, 0.9999 and 0.99999 seem to be only 0.00009 apart, so that 0.502 is much further away from 0.503 than 0.9999 is from 0.99999. To get to probability 1 from probability 0.99999, it seems like you should need to travel a distance of merely 0.00001.
> But when you transform to odds ratios, 0.502 and 0.503 go to 1.008 and 1.012, and 0.9999 and 0.99999 go to 9,999 and 99,999. And when you transform to log odds, 0.502 and 0.503 go to 0.03 decibels and 0.05 decibels, but 0.9999 and 0.99999 go to 40 decibels and 50 decibels.
> When you work in log odds, the distance between any two degrees of uncertainty equals the amount of evidence you would need to go from one to the other. That is, the log odds gives us a natural measure of spacing among degrees of confidence.
[1]: https://www.lesswrong.com/posts/QGkYCwyC7wTDyt3yT/0-and-1-ar...
Edit: formatting
Your example shows that (non-linear) transformations aren't linear, and that our gut feels about likelihood aren't either, which is true enough.
However, you wouldn't say "101 ˚C isn't a temperature" because it takes much less energy to heat a wet thing from 97 to 99˚ than it does to bring it from 99 to 101˚ because of the phase change.
Of course, they are valid values for the set of probabilities, just like 0K is a valid temperature, and c is a valid speed for massive things. You just will never see any of those.
Sure you can: if you are using inferential statistics and trying to determine the population incidence of some trait and every sample has a sample incidence of exactly 1 then the population estimate will be exactly 1. And that works the same way for 0, or any other value, too.
> c is a valid speed for massive things.
No, it's not. An object with any rest mass would require infinite energy to reach c. It's an excluded upper bound, not a valid speed.
I put 100 lb. of potatoes in the sun. The potatoes were 99% water by weight. After drying a while, the potatoes were 98% water. How much did they weigh at that point?
They offer a series of actions that let you lose or gain x% n many times in a row, with an equal chance of each. The fact that +x% doesn't cancel out a -x% compounds on the difference in lay person expectations to produce horrific odds for microtransactions.
Certainty has nothing to do with how many people said yes or no, it’s about how many people are in the experiment.
If you sample size of 2, and both like the new UI, then you can’t say you have 100% certainty.
I.e. 3/4 people isn’t the same certainty as 75/100
I think the point he’s trying to make is it’s easier to get the statement “95% of people prefer X” from a study then to get “99% of people”. The first requires >20 people, the later >100.
But he’s tying together the concepts of sample size, confidence, and the outcome of the experiment in a way hat doesn’t make sense
At 95% certainty, you have 95 people saying "yes" and 5 people saying "no"?
So it's easier to make an apples to apples comparison? What point is the author trying to make with changing the scale?
> It feels like a difference of four people when, in reality, it’s a difference of 80. That’s a much bigger difference than we expect.
I had to stop reading here.
Your mind sees the 5% going to 1% (or the 95% going to 99%) and thinks it's a small difference. When in actuality it's a big change.
So, what do you want to know, "yes for every 100" or "yes for every no" ? It matters.
What youre struggling with is the counterintuitive nature of applied statistics vs pure math, and this is the point TFA was trying to make.
> You aren’t getting 80 more signups as suggested
TFA isnt saying you "get" more, but just illustrating how different 95 and 99% actually are. Its restating the potato paradox, linked elsewhere in the thread
I wasnt aware "TFA" could be interpreted with a negative connotation, although that seems obvious in retrospect. Just trying to participate in the HN community, and also because its less typing : )
As a matter of writing style, repeating acronyms are invisible to the reader, whereas repeating words are annoying and remove value. This may be an opinion I gained from my military service, which was acronym-heavy.
I appreciate the question and its perspective.
But "TFA", although it derives from "RTFA", never seem to had the same negative connotations. It's just that sometimes you want to refer to the original article in question, but "original article" is long to type and/or ambigious. (Do you mean the news article from the NYT, or the scientific paper the NYT article is reporting on?) And "TA" is too short for people to clearly know what you're talking about (And did you mean "teaching assistant"?) "TFA" is short and unambigious: it always means the article linked to from the main page.
Long story short: Although etymology would suggest that "TFA mentions this" is as aggressive as "Maybe you should RTFA", in actual developed usage, they're very much not the same.
That was my understanding and my intended usage.
An 80% decrease in missed signups only causes about a ~5% increase in revenue. That’s an important point if the team that produced that 5% revenue increase costs a large amount of money to run. At close to a million a year for the team, that’s only going to be worth it for some.
And that was the whole point of the article: this optimization usually isn’t worth the cost.
That's exactly the point. How many people said yes for every one who said no:
50% -> 1 yes for every no
95% -> 19 yeses for every no
99% -> 99 yeses for every no
99.9% -> 999 yeses for every no
This is why you should never stop an A/B test once you've "hit your statistical significance". Always choose the number of tests you'd need to prove the significance before you start, and let it run even if it's "obviously winning" (or losing).
>At 5% uncertainty
>At 1% uncertainty