Hacker News Reader

Comment by searealist | Hacker News Reader

So basically, your claim is that a normal distribution is less random than a uniform distribution? Can’t say I agree.

If you order randomness by entropy, the uniform distribution is indeed, in some sense, "more random" than any other probability distribution with the same support.

As I understand it, the claim is that the volume being added is tiny compared to the real volume.

Well, by definition it is. It’s why we have names like normal and uniform.

By definition it's not. It's just as random. These names just tell you a few key properties of the distribution.

That’s a bit like saying that our bodies are random numbers because of quantum fluctuations. It may be true, but the distribution is the part that matters.

It’s still ridiculous to use a normal distribution to calculate how much money should go into an insurance fund that needs to cover sudden, severe losses.

Would you say that a function that always returned 0 or 1 would be more/less/equally random than a function that returned any int32 with equal probability?

I would normally say the latter is more random, because you're getting more entropy per call. In the same sense, a function drawing from a normal distribution is less random than one drawing from a uniform distribution.

(All of this depends on a computers view of the world, where values are in data types that have some number of bits. In a mathy view of the world where every value is effectively infinitely many bits you get just as much entropy sampling from a normal distribution as a uniform one.)

> a function drawing from a normal distribution is less random than one drawing from a uniform distribution

That's some mathematical nonsense.

A normal distribution is continuous from negative infinity to positive infinity, so even by your flawed judgement of what makes something "random", a sample from the normal distrubtion has far more possibilities.

The standard normal distribution is almost the definition of random.

Again, it depends on whether you are looking at this from a pure math perspective or a "math as implemented on computers" perspective. I'm only claiming that this use of more/less random makes sense in a computational context.

But in that context, there is more entropy in a float32 drawn from a uniform distribution than a float32 drawn from a normal distribution, no?

Actually, I'm not sure this is mostly about a mathematical vs computational perspective. Let's look at how entropy is defined.

It's not just the number of possibilities, it's how likely they are. For example, consider two functions:

    A: 1% chance of 1, 1% chance of 2, 98% chance of 3
    B: 50% chance of 1, 50% chance of 2

The entropy of a distribution, in bits, is:

    H = - sum over all possibilities of (p * log_2 p)

So for A it is:

    H_A = -(2 * 0.01 * log_2(0.01) + 0.98 * log_2(0.98))
        = 0.16

And for B it is:

    H_B = -(2 * 0.5 * log_2(0.5))
        = 1

So you can see that A has lower entropy than B, because it's more concentrated.

For continuous distributions this is defined with an integral instead of a sum, but it's the same idea:

    H = - integral from -inf to + inf of (p * log_2 p)

Whether there's more entropy in a sample drawn from a uniform or normal distribution, then, depends on how wide your distributions are. Consider:

   C: uniform distribution from -1000 to +1000
   D: normal distribution centered on 0 with standard deviation 10
   E: uniform distribution from -1 to 1

We then have:

   H_C = - log_2(2000) / log_2(2) = 10.97
   H_D = 1/2 (log_2(2pi * 10^2) + 1) = 5.15
   H_E = 1

Note that continuous and discrete entropies mean different things, and their values can't be compared directly. That H_E and H_B are both 1 is not meaningful.

I think you've changed what you're meaning by entropy, it isn't really a meaningful measure of "randomness".

Afterall, how can U(0,1) be considered to be "less random" than U(0,1000) when U(0,1000) = 1000*U(0,1). A constant factor can't change how "random" something is.

If you look at this from a computers ("everything is discrete eventually") perspective, U(0,1) is less random than U(0,1000) because U(0,1000) != 1000U(0,1). For example, if you're using a fixed-point encoding that allocates values uniformly, it will have 1000x as many possible values for [0,1000] as [0,1], and so getting any specific value from U(0,1) is 1000x more likely than any specific value from U(0,1000). If you draw from U(0,1) and then multiply by 1000 you'll never produce most values that U(0,1000) could produce.

If you look at it from a math ("continuous is actually possible") perspective I have a much weaker handle on it, but I think the problem is that the entropy of 1000U(0,1) is not the same as the entropy of U(0,1)?

> just as random

Let’s do a lottery together. We each throw in some money. Since they’re both just as random, I’m sure you won’t mind if I use a normal distribution, and choose each of our positions on the number line. :)

Your assertion is missing some needed qualifiers.

You've probably just got different understandings of the term "random."

> Well, by definition it is.

If given only mean and standard deviation, the normal distribution is the maximum entropy distribution i.e. maximally random distribution.

Reply on news.ycombinator.com