Ooops, I forgot to include something in the list of conditions. For every probability I mentioned in the post you just quoted and its grandparent post, we should add to the list of conditions that the number of samples used is the same as the number of samples used in our experiment.

Anyways, if it sounds like nonsense then I will try rephrasing. Right now the threshold for statistical significance is p = 0.05, sometimes lower depending on the field. Let’s say we want to lower this threshold, and we need some way of determining how low it should be. I am suggesting that this could be done by deciding on an acceptably safe threshold, say q, for p(null | T, num samples, normal, iid), and an acceptably safe upper bound, say x_0, for p(null | num samples, normal, iid) / p(T | num samples, normal, iid). Then we use q / x_0 as the threshold for statistical significance.

We would also be using p-values for a null hypothesis corresponding to a range of means rather than a single value (or a range of differences in means, or something else suitably adapted to the type of test we are doing), because with a single value, having an upper bound on the probability of the null hypothesis is vacuous, as you very eagerly point out.

You could argue that this is useless because what we really want to bound is p(null | num samples, T), not p(null | num samples, T, normal, iid). I would respond that, yes this would be the more desirable quantity to know, but we need to make some simplifying assumptions in order to make this problem tractable, and while the normal and iid assumptions probably aren’t true, there are at least situations where we can be confident that they approximate reality reasonably well.