Hacker News Reader

Comment by tprice7 | Hacker News Reader

You mean for computing the p-value associated with a range of means rather than a single null mean value? That’s the only thing I can think of for which providing code or calculation steps would be applicable (well, there’s also getting a Fermi approximation for the total number of experiments ever performed in a discipline, which I gave as a preliminary suggestion for a cautious estimate of x, but I don’t have time to do that and the main difficulty of that isn’t the math anyways). Anyways I would be happy to provide that if you can confirm that this is what you meant and that you are asking out of genuine curiosity.

View on HN

>"our upper bound on p(null | T, normal, iid) small for significant p-values (i.e. p(T | null, normal, id))"

This sounds like nonsense to me so I would like to see an example of what you mean.

Ooops, I forgot to include something in the list of conditions. For every probability I mentioned in the post you just quoted and its grandparent post, we should add to the list of conditions that the number of samples used is the same as the number of samples used in our experiment.

Anyways, if it sounds like nonsense then I will try rephrasing. Right now the threshold for statistical significance is p = 0.05, sometimes lower depending on the field. Let’s say we want to lower this threshold, and we need some way of determining how low it should be. I am suggesting that this could be done by deciding on an acceptably safe threshold, say q, for p(null | T, num samples, normal, iid), and an acceptably safe upper bound, say x_0, for p(null | num samples, normal, iid) / p(T | num samples, normal, iid). Then we use q / x_0 as the threshold for statistical significance.

We would also be using p-values for a null hypothesis corresponding to a range of means rather than a single value (or a range of differences in means, or something else suitably adapted to the type of test we are doing), because with a single value, having an upper bound on the probability of the null hypothesis is vacuous, as you very eagerly point out.

You could argue that this is useless because what we really want to bound is p(null | num samples, T), not p(null | num samples, T, normal, iid). I would respond that, yes this would be the more desirable quantity to know, but we need to make some simplifying assumptions in order to make this problem tractable, and while the normal and iid assumptions probably aren’t true, there are at least situations where we can be confident that they approximate reality reasonably well.

Also, as an aside, I’m not sure what you mean specifically when you say “model”. The usual definition is a set of probability distributions on a common sample space. But when you say things like “all models are wrong”, I’m assuming you are just using “model” as a synonym for “distribution”, because this claim is vacuously false using the usual definition of model (just take the set of distributions to be every distribution on that sample space; this must contain the true distribution by definition). But then when you say “Looking closer at this you are describing one model but testing different values of one of the model parameters”, this seems to make more sense if you are using the usual definition of model. So I’m not sure what definition you are using.

Also, in case this clears up some confusion, I was using the nonstandard definition in some posts because I inferred that this was the definition you were using.

Here is an example of a model.

The other day I got a robocall from one of those spoofers that uses the same area code + three digits as the number being called. This call happened to be a number I did know. Did they just happen to hit upon this number, or is something more sinister going on (eg using a hacked address books)?

Say I've gotten nCalls calls like this so far, whats the probability at least one of them would be from a number I know?

The probability of any given number being used will be 1/9999, if I know 5 numbers with the same first digits as my own it would be 5/9999, etc. This leaves 1 - nKnown/9999 other possible numbers to be used. The probability they keep using unknown numbers will then just be eg, (9998/9999)^nCalls and we take 1 minus this value to get the probability that at least one of the calls will be from a known number. Here is the model:

  model_1 = 1 - (1 - nKnown/9999)^nCalls

Lets say I've gotten about one call like this every day for the last two years. So nCalls = 730, and I know 3 numbers that share the same digits including my own (I am assuming they also spoof the number they are calling). Then the probability of getting at least one call from the known numbers would be 20%. I made simulation in R and see the same results:

  # Returns percent of calls coming from a known number for nSim experiments
  sim <- function(nSim, nCalls, nNum, nKnown, replace = FALSE){
    res = replicate(nSim, sample(1:nNum, nCalls, replace = replace) %in% 1:nKnown)
    return(colMeans(res))
  }

  model_1 = mean(sim(1e4, 730, 9999, 3, T) > 0)

I would guess that robocallers are pretty cheap so are probably going with the simplest approach possible. But perhaps they are slightly more advanced and they avoid reusing the same number (ie, if it went unanswered and I never answer unknown numbers) and avoid using the callees number. This can be done by changing a few arguments in the sim:

  model_2 = mean(sim(1e4, 730, 9998, 3, F) > 0)

The two models predict nearly the same thing in the range of 730 calls, so p(data|model_1) = p(data|model_2) = 20%. I think the simpler model is still more probable though, so lets say p(model_1) = 0.75 and p(model_2) = 0.24 and p(model_x) = .01. Model x is that something more shady like using the hacked contact info is going on, this is pretty vague and can explain anything so give p(data|model_x) = 1.

Then we use Bayes' rule:

  p(model_1|data) = .75*.2/(.75*.2 + .24*.2 + .01*1) = 72%
  p(model_2|data) = .24*.2/(.75*.2 + .24*.2 + .01*1) = 23%
  p(model_x|data) = .01*1 /(.75*.2 + .24*.2 + .01*1) = 5%

EDIT:

Actually, I made an error for model 2. Since I am not including my own number, nKnown should be 2 instead of 3.

    model_2 = mean(sim(1e4, 730, 9998, 2, F) > 0)

This gives p(data|model_2) = .14, and:

  p(model_1|data) = .75*.2 /(.75*.2 + .24*.14 + .01*1) = 77%
  p(model_2|data) = .24*.14/(.75*.2 + .24*.14 + .01*1) = 17%
  p(model_x|data) = .01*1  /(.75*.2 + .24*.14 + .01*1) = 5%

Reply on news.ycombinator.com