Hacker News Reader

Comment by runarberg | Hacker News Reader

Honest (possibly stupid) question: Do people working on statistical and probability application never need a numeric type optimized for probability values (i.e. values between 0 and 1 that can get really close to either; for example the really almost certain value of p = 1 - 10^-13)? In the same way that financial application needs a decimal type?

I’ve only worked on probability applications at a surface level for a tiny bit and never really needed it, but I kept wondering whether there were such a numeric types, and if not, what people did if they needed it.

View on HN

Log probabilities are used a bit for this, and most numeric packages (numpy for instance) have special functions for computing log(1+x) which are more accurate when x is tiny.

This would be really nice, but the only solution I know here involves working with numbers that represent a function of p rather than p itself. And what you do depends on what you're computing. In general, you'll find lots of useful math functions that have specialized routines around zero and one, like numpy's log1p function, logaddexp, expm1, and company.

Log odds ratios are also good representations that have this kind of accuracy between zero and one. They're good when you need them. (e.g. 100:1 odds are log(100/1) and 1:100 odds are log(1/100))

I see. So basically you just know a bunch of log rules and then map your distribution into a log-distribution. And I guess this becomes second nature to a seasoned statistician. So a dedicated numeric type for probability value is nothing more than a nice-to-have.

Kinda, but to make it more explicit: with floating point numbers you don't even need the explicit log distribution because it's kinda baked in already, with the number split between an absolute and an exponential part. So floats capture an exponential scale of precision naturally, allowing you to write 1e-200 and 1 and 1e200 just as easily, with a common relative precision around each. You use the log scale as a mental model and then just keep floating point issues in mind while you work.

The handy functions are more about making sure that cancellations don't happen by providing a few primitive operations that cover lots of the uses. Like the handy log1p which computes log(1+x). If you did this with the log function, you'll compute log(1+1e-200)=log(1)=0. If you use log1p, you get log1p(1e-200)=1e-200. You avoid the loss of info that comes with adding something close to zero to something that isn't close to zero, which is the real trick. And then if you need something close to one, you probable just use the converse: instead of using p=1-eps, you just work with eps itself.

Reply on news.ycombinator.com