Why is everything based on likelihoods even though likelihoods are so small?

It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions.

I think the answer is:

Yes, individual likelihoods are so small, that yes even a MLE solution is extremely unlikely to be correct.

However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using.

Much like how the average is unlikely to be the exact value of a new sample from the distribution, but it's a good way of describing what to expect. (And gets better if you augment it with some measure of dispersion, and so on). (If the distribution is very dispersed, then while the average is less useful as an idea of what to expect, it still minimises prediction error in some loss; but that's a different thing and I think less relevant here).

They are useful because the integral of the likelihoods is not infinitesimal.

The probability that your yard stick measures 1.0000000000000 yards is basically zero, but the probability that it's within one inch of that is close to one.

We generally prefer to use probability density functions with the property that most of the probability density is close to the maximum likelihood.

So, in the yard stick example, the yard stick lengths are probably gaussian, so if check enough lengths, you'll get a mean (== length with the maximum likelihood) that approaches 1.00000000000 yards (you'd hope) with some small standard deviation (probably less than an inch).

Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million.

Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million.

That is, the sequence we observed supplies about twice as much evidence in favor of bias = 0.5 as compared with bias = 0.4—this is likelihood.

Likelihood ratios are all that matter.

Morals:

- The more complex the event you're predicting (the rarer the tyical observed result) the smaller the associated likelihoods will tend to be

- It's possible that every observed result has a tiny probability under every model you're considering

- Nonetheless it makes sense to use the ratios of these numbers to compare the models

- This has nothing to do with probability densities or logarithms, though the fact that we often work with densities also makes absolute likelihood values relative to the choice of units

Added in edits:

- You could summarize the sequence with the number of heads or tails and then the likelihood values would be larger but the ratios would remain the same (it's a sufficient statistic). Similarly in the CrossValidated question one could summarize the data with the mean and sum of squares. But this doesn't work in general, e.g. if we have i.i.d. draws from a Cauchy distribution.

My reply was, “it happens to every user, the first time they use the app.” And then something about how frequency has nothing to do with it. Every single user was going to encounter this bug. Even if they only used the app once.

I already had a toolbox full of conversations about how bad we are at statistics, but that one opened up a whole new avenue of things to worry about. One that was reinforced by later articles about the uselessness of p95 stats - particularly where 3% of your users are experiencing 100% outage.

But the one that is more apropos to the linked question, vs HN in general, is how people are bad at calculating the probability that “nothing bad happens” when there are fifty low probability things that can go wrong. Especially as the number of opportunities go up.

And the way that, if we do something risky and nothing bad happens, we estimate down the probability of future calamity instead of counting ourselves lucky and backing away.

The center of a normal distribution has high likelihood (e.g. 1000000) if the standard deviation is small or low likelihood if the standard deviation is large (e.g. 1/1000000.)

This effect is amplified when you are working with products of likelihoods. They can be infinitesimal or astronomical.

Giant likelihoods really surprised me the first time I experienced them but they’re not uncommon when you work with synthetic test data in high dimensions and/or small scales.

They still integrate to the same magnitude because the higher likelihood values are spread over shorter spans.

In this case maximum likelihood approximate bayesian estimation with a mostly reasonable prior. Furthermore you could look at the convergence properties which are good.

You could probably design some degenerate probability distribution that ml-estimation behaves really badly for, but those are not common in practice.

Hence why using A and THE is important.

If I look at coin flip data, I know the data comes from a coin flip, but any specific count of heads vs tails becomes less and less likely the more flips we do. So likelihood being small tells us nothing on its own.

The value of likelihood comes from the framework you use in. If I wanted to make a best guess at what the balance of the coin is then I could find the maximum of the likelihood over all coin balances to get the most representative version of my model. Similarly, I can compare two specific coin biases and determine which is more likely, but that alone can’t tell me anything about the probability of the coin being biased.

In practice, that means that if you have an alternate "non-random" or "less random" explanation for the data, you'll be convinced that it's almost surely the correct one after just a few samples (via the obvious Bayesian decision framework, or just "common sense").

For example, imagine that you are rolling a die and 3 always comes up. On every roll, the likelihood of the die being a fair random die (as opposed to a loaded die) is divided by the number of sides, so with a 6-sided die, you'll usually be convinced that it's loaded after just 3-20 times of giving the same result (depending on your prior on it being loaded vs fair and your decision threshold).

Likewise, if only 1 and 2 come up you'll quickly be convinced that it's an unusual die that only has 1 and 2 symbols on the face.

Or another way to look at it is that processes are usually not random at all (rather they are usually deterministic, but the initial state is unknown) so a random distribution is a very bad model, and having any information about the initial state at all will drastically increase the likelihood and thus make the model with information strongly preferred; the likelihood of the random model is so low because that model is very bad, even though it may be the best available.

This tangent may have been unrelated or irrelevant but I’ve long concluded that in practice statistics is far less solved than people might expect if they’ve never had to reproduce any of the numbers given to them by statistical analysis

“It appears to be very unlikely in a certain sense that this comment is written in English. Yes, this sequence of characters is much more likely in English than in French. But you can’t even fathom how unlikely it was to be ever written in English!”

What I do have a problem with is lack of conceptual framework for dealing with small probabilities of real life events.

For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times?

In this case the question then becomes "Why are we looking at energy if at most ordinary temperatures systems aren't anywhere near 0 energy?". The overly simplified answer is "Because bowling balls still roll down, even at room temperature."

Which is actually kind of interesting to think about. Why do bowling balls roll down when they have vastly more thermal energy than gravitational potential? To get an answer you have to invoke the second law of thermodynamics. Which is annoyingly a bit hard to really get to understand. In short it implies that energy likes to become more disorganised, so something like gravitational potential which is very organised will eventually devolve into heat which is (by definition) disorganised. So bowling ball roll down.

You can play perfect blackjack and card count at a table with good rules and lose plenty because your advantage is small (< 2%) and your repetitions are limited.

Statistics get even worse when the probabilities are chained because the weakest estimator bounds the rest.

Essentially, if you always follow statical advice you should do better than average, if you're lucky. There are better heuristics than statistics in most fields of human decision making.

The mathematical and computational reasons that the logs are smaller and easier to read in a plot.

Maximum likelihood estimation has some weird cases when the distribution is not "well behaving".

Is that men experience much that is improbable.

- Fragment of a lost play by Agathon, ca. 2,400 years ago, quoted in Aristotle's The Art of Rhetoric

> As we can see, even from the correct distribution, the likelihood is very, very small. Thus, it appears to be very unlikely in a certain sense that these numbers came from the very distribution they were generated from.

The person who asked the question is simply confused between likelihoods and posterior probabilities. The likelihood of d values from a Normal Distribution is defined to be the probability of sampling those d values given the parameters of the Normal. It is not the probability that those numbers came from that Normal Distribution. To answer the latter question, you need to say what other possibilities you're considering (perhaps some other parameter values) and use Bayes Rule. The other answers mention that ratios will be involved, but the way to see why ratios are involved is to look at Bayes Rule.

We're talking about the whole likelihood surface here, not just the single point that's the maximum likelihood estimator. The MLE is a method for choosing a valid point estimator from the likelihood function; it has some good properties, like being consistent (if you have enough data it converges to the truth) and asymptotically efficient (converges smallest possible variance) so long as some criteria are met.

But the MLE is not the only choice; for any given model, other procedures can be admissible estimators https://en.wikipedia.org/wiki/Admissible_decision_rule - it's just they also have to be procedures based on the likelihood function. In other words, your procedure doesn't have to be "take the likelihood function and find its maximum" but it has to be "take the likelihood function and... do something sensible with it."

So the MLE is popular in the frequentist world where you have to make the decision rules using the likelihood directly; in the Bayesian world, you take the likelihood and combine it with a prior, to make an actual probability distribution. Then you get things like like MAP (mode of the posterior) or the Bayes estimate (expectation of the posterior) - alternatives to MLE that still use the likelihood surface.

Of course this all works only if the underlying probabilistic model is literally true. So, in the machine learning world where the models are judged on being useful on usefulness and not expected to reflect mathematical reality, you're allowed to do things inconsistent with likelihood principle, like regularization tricks. In some physics situations (astronomical imaging comes to mind) where the probability model really is governed by the rules of nature, sticking to likelihood principle actually matters.

As to the question of being small, well, the likelihood is the probability (density) of the exact data you observe given parameters. Let's say you know the true parameter (the mean and standard deviation) and you observe a thousand draws from a normal distribution. Of course the probability of observing the very same pattern of a thousand values again is overwhelmingly unlikely. But if the mean was way different, that pattern would be proportionally even more unlikely. We should only care about relative probabilities. What's the probability that the universe evolved in exactly such a way that your cat will have exactly this fur pattern? Astronomically small. What's the probability that the universe evolved in such a way, and some of that fur ends up on your furniture? Another unimaginably small number. But what's the probability that, in a universe where you and your cat exist as you are, his fur will get everywhere? That's pretty much a certainty.

Think about it. That distribution is continuous over an infinite amount of numbers. If you select any number the chances of that number being generated will be essentially zero. According to the theory there is no possibility for any number on the distribution to be generated. This is correct.

Yet when you use the random number generator you get a number even though that number technically is impossible to exist due to zero probability. Does this mean there is a flaw in the theory when applied to the number generated?

Yes it does. The theory is an approximation of what's going on itself. No random number generator in a computer is selecting a number from a truly continuous set of numbers. It is selecting it from a finite set of numbers from all available numbers in a floating point specification.

Even if it's not a computer when you select a random number by intuition from a continuous distribution you are not doing it randomly.

Think about it. Pick a random number between 0 and 1. I pick 0.343445434. This selection is far from random. It is biased because there is an infinite amount of significant figures yet I arbitrarily don't go past a certain amount. I cut off at 9 sigfigs and bias towards a cutoff like that because picking a random number with say 6000 sigfigs is just too inconvenient. You really need to account for infinite sigfigs for the number to be truly random which is impossible.

So even when you pick numbers randomly you are actually picking from a finite set.

In fact I can't think of anything in reality that can truly be accurately described with a continuous distribution. Nothing is in the end truly continuous. Or maybe it does exist, but if it does exist how can we even confirm it? We can't verify anything in reality to a level of infinite sig figs.

If R was accurately calculating likelihood it should give you zero for each number. And the random number generator should not even be able to exist as how do even create a pool of infinite possibilities to select from? Likely R is giving some probability over a small interval of numbers.

That's where the practicality of the continuous distribution makes sense when you measure the probability of a range of values. You get a solid number in this case.

Anyway the above explanation is probably too deep. A more practical way of thinking about this is like this:

It is unlikely for any one person to win the lottery. Yet someone always wins. The probability of someone winning is 100 percent. The probability of a specific someone winning is 1 over the total number of people playing.

Improbable events in the universe happen all the time because that all that's available. It's highly improbable for any one person to win the lottery but if someone has to win, then there is a 100 percent chance that an arbitrary improbable event will occur.

This is more easily seen in a uniform discrete distribution rather then the normal continuous distribution.

In the case of the normal distribution it is confusing. In a normal distribution It is far more likely for an improbable event to occur then it is for the single most probable event to occur.

Think of it like this. I have a raffle. There are 2 billion participants. Each person has one ticket in the bag, except me. I have 100,000 tickets in the bag I am the most likely person to win.

But it is still far more likely for anyone else but me to win even when I am the most likely person to win. An arbitrary improbable event is more likely to occur then the single most probable event.