Why, if p=.04, we can't say the study has a 4% chance of being false positive
twitter.com
twitter.com
for increased readability.
He starts off with "Imagine a world of scientific hypotheses" but isn't that world infinite? I could ask whether 1 is a number, 2 is a number, 3.14 is a chicken, etc. Aside: I would argue that the cardinality of those infinites are actually the same since you can form a bijection between true and false hypotheses just by negating the statement (1 is a giraffe, 1 is not a giraffe).
So I assume he means the finite world of all hypothesis that scientists(which are finite since the universe will eventually end) would consider interesting enough to test? Even then it doesn't seem narrow enough. The pool of hypotheses has a shifting distribution based on time period (e.g. 1800s vs modern science). And you can further slice them down field, journal, affiliation, exact subject, etc.
If that's being pedantic, ignore defining what the world of hypotheses looks like for now. Even given a fixed world, how can you estimate the true number of "True Hypotheses"? You only really see the end result of your study (positive or negative) and you probably don't even know about the existence of most of the negative studies.
In short, still super confused! Links to intro to p-values/stats appreciated.
Short version; the p-value is only sufficient to disprove a hypothesis, not to prove it. If your p-value is 1% than that means you make experiments until the probability that you are wrong is less than 1%. (for example, you throw coins 100 times, 90 times is head, the probability of that result is below 0.04 so the coin is not fair)
That still doesn't mean you are right. Just that at the moment you have not been prove wrong or the data is not sufficient to prove you wrong.