Paradoxes of probability and other statistical strangeness | Hacker News Reader

Paradoxes of probability and other statistical strangeness

theconversation.com

132 pointsseycombi106 comments

A less popular but perhaps more influential phenomenon is Stein's Paradox [1]. Here's a provocative example often given to illustrate it: Say you have a baseball player, soccer player, and football player, and you wish to estimate the true mean number of home runs, goals, and touchdowns each scores per year. If you have their last ten seasons worth of data for each, then the obvious thing to do, for each player, is to estimate the true yearly mean score for each player by their average yearly score from the last ten years. (E.g., the baseball player hits an average of 20 home runs each year, so let's estimate their true mean yearly home runs by 20). Stein's Paradox says that you can actually do a lot better than this.

Even more crazy, the James-Stein Estimator which does this actually uses data about the football player and soccer player to make predictions about the baseball player, (and vice-versa). This is deeply unintuitive to most people since the players aren't related to each other at all. The phenomenon only holds with at least three players; it doesn't work for two.

(More generally, Stein's Paradox is the fact that if you have p >= 3 independent Gaussians with a known variance, you can do better in estimating their p-dimensional mean than just using their sample means).

I've spent a bunch of time trying to understand why this actually works [2]; to be honest I still don't deeply understand. But nonetheless the consensus is that the same shrinkage phenomenon is what causes improved performance for a variety of high-dimensional estimators, (lasso or ridge regression, e.g.), making the paradox very very influential.

[1] https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator [2] https://www.naftaliharris.com/blog/steinviz/

Well this does make a little bit of sense if after estimating things get worse for a particular case.

If MSEs were 5, 5, 100, and after they are 10, 10, 80, the total got better but the prediction for the other two got worse.

Yeah, reading through the Wikipedia, it looks like this reduces the total error of the combined estimator, but the error compared to an estimator of any one single parameter could be worse. So you can combine whatever crazy parameters you want, but it's only really relevant when you have things that are associated with each other somehow, and you want to reduce the total error of estimating all of them.

> it's only really relevant when you have things that are associated with each other somehow

The proof of lower overall MSE assumes the variables are independent.

This is not why it works. The James-Stein estimator applies in the case of independently distributed Normal variables with equal variance, so the individually optimal estimators for each parameter have the same MSE.

I think it is impossible James-Stein estimator of the whole sample can outperform all of the three MLE for each particular. Not that I would know of a way to generate a particular from a joint estimator.

I think I may have misinterpreted your original comment. It's true, as you say, that in the Stein estimator some individual MSEs get worse to yield a lower overall MSE. I was focusing on your example and assuming you meant this was only possible because some individual MSEs were larger than others to begin with (under the individually optimal estimator).

Not sure I understand. Why should the number of home runs per year, the number of goals per year, and the number of touchdowns per year have equal variance?

You probably wouldn't expect them to. But the same kind of Stein phenomenon holds under a much broader set of conditions, including arbitrary covariance matrices and arbitrary quadratic loss, (see e.g. [1]). It's a very general phenomenon!

[1] https://projecteuclid.org/euclid.aos/1176345691

I'm going to go out on a limb with my intuition, and hypothesize that the underlying premise of the method zeroes in on a common characteristic of games humans tend to enjoy.

A universe of people wouldn't find these games generally interesting, if they didn't present outcomes above a certain threshold of unexpectedness. The underlying rules of each game are tuned into the equipment used, and a balance is reached, where game play is fair, but still requires players to develop skills.

Because each sport adheres to the premise of capturing interest in players and spectators, they all present the same scoring tendencies, when aggregating and generalizing.

If you change the motive of the activity (mix games with non-games), and the artifact that represents success (mix freely tallied points with rare physical tokens or discovered evidence), so that the behaviors being compared are dissimilar, the predictions will become unreliable.

For example, when comparing "victories" across lawyers, geological prospectors, and sports players you probably would not be able to make predictions about all, by lumping each area's statistics together. A gold-mining prospector probably wouldn't encounter success in the same way a trial lawyer would, and neither would help you predict or generalize a hockey game.

But, an oil driller, a diamond prospector, and a gold prospector would likely compare, based on the geological goal sought. A forensic analyst, a private detective, and a trial lawyer might compare, also, based on the human factors of investigation. And, thus, so too, with sports where freely tallied points measure a player's skill at achieving an event in game play.

I don't understand. If the average of the last 10 seasons is 20 home runs, what would be a better predicted value? You are a bit short in explaining here?

Your site, and the Wiki link, is very math formular heavy. Is there an explanation for someone who forgot all his statistic courses and greek letter thingys?

This is maybe a better explanation:

https://jmanton.wordpress.com/2010/06/05/comments-on-james-s...

It's still math heavy, but there is some explanation. It's hard to explain without the math, since the math if fairly integral to it, that's why it's such an amazing discovery. My understanding is that it's saying that the variables are independent, but the measurement is not. So in the case of the athletes, it's not that home runs predicts touchdowns or goals, but that by using a Stein Estimation we would get a more accurate measure of all three in aggregate. The example used in the article is less interesting, but probably better for understanding:

For example, if i=1,...3 represents the financial cost of claims a multi-national insurance company will incur in the next year in three different countries, the company may be less concerned with estimating the values of the individual means accurately and more concerned with getting an accurate overall estimate.

Here's my intuition. Let's say you have 1000 coin flippers. They flip a coin 10 times, and none of them has any special powers, and the coin is fair. Some of them will get an equal number of heads as tails, but there's a good chance you'll get tsome who get 9 or 10 heads, and also some who get 9 or 10 tails. As the probability to get 10 heads in a row is 1/1024, if you see one or two guys how get only heads, or only tails, you will attribute that to the natural variability of the outcomes.

Now imagine that these are not coin flippers, but some guys who have some skills to do something, but the outcome has a large variability nonetheless. For example running backs in the NFL league. There are running backs (RB) who average 2 years per carry (ypc), and others who average 5. 5 ypc is stellar by the way, 4 is very good, 3 is decent, and 1 or 2 not so much. But obviously, RBs get a different yardage for each carry. Now, let's say you follow the first 4 games of the season and get the average ypc for each RB. You would like to predict for each RB the average ypc for the rest of the year. The classical statistical estimation is that the current average is the best estimator for the future average, but from the extreme example with the coin flippers above, we know that this is not quite the case. Using a bayesian estimation, we get that a better estimator is if we move the current average towards the overall mean. This is called a shrinkage or James-Stein estimator. In the case of the coin flippers, you move the average all the way to 1/2, and that estimator is correct. In the case of the running backs, you don't shrink that much, and it's a cute exercise in math to see how much you shrink if you assume some distributions around the overall ypc for RBs in the league and around the ypc of an RB given his average ypc.

If you want some further intuition, think of the Sports Illustrated curse. It was observed that NFL players who make it to the cover of the SI magazine are generally "cursed", i.e. they don't do as well after as they did before. One amusing case is the (former) New England Patriot Jonas Gray, who made the cover of SI after a phenomenal game with the Indianapolis Colts in 2014 (201 rushing yards, 4 touchdowns), but then he showed up late to work and was promptly benched for the rest of the season. Generally though, players don't do anything stupid like that, but simply "regress to the mean". That regression to the mean is what explains the shrinkage estimator, and the Stein paradox.

> If the average of the last 10 seasons is 20 home runs, what would be a better predicted value?

You are correct, 20 is the best estimate for this single variable (or similarly for any single variable in isolation).

Only if the objective is to minimize the total MSE (Mean Square Error),

          (Ph - h)² + (Pg - g)² + (Pt - t)²
    MSE = ---------------------------------
                          3

then it pays off to bias each estimate – Ph, Pg, Pt – slightly towards zero. If any of the observed values is larger than the true value, we do improve the estimation by using a correction coefficient slightly under 1. If the observation happens to be smaller than the true value, we do make a mistake. But we make a smaller mistake when the observed value was small because it was small, than what we improve when it was large. A set of 3 independent variables is already large enough that this gamble pays off in average (in the combined total error of the 3 estimates).

My favorite probability paradox has always been the Monty Hall problem[1]:

Suppose you're on a game show, and you're given the choice of three doors:

Behind one door is a car; behind the others, goats.

You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat.

He then says to you, "Do you want to pick door No. 2?"

Is it to your advantage to switch your choice?

[1] - https://en.wikipedia.org/wiki/Monty_Hall_problem

I love this one, but I guess it must be less common to have such a problem in the real life. Or I'm wrong?

As the Wikipedia article points out, that problem as stated is ambiguous. You need to know under what circumstances the host would have opened that door. The standard assumption is that the host always reveals a door with a goat behind it.

However, another possibility is that: (i) if both of the remaining doors contain goats then the host reveals one of them; but (ii) if one of the remaining doors contains the car then the host doesn't open a door at all.

That variation doesn't make much sense as a paradox. If (i) you know that you should stay 100% success. If (ii) you know you should switch, 50% success.

If you see this problem as a special case of another problem the answer becomes pretty intuitive.

Suppose there are N (let's pick 1000) doors: Behind one door is a car; behind the others N-1 (999), goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens N-2 (998) doors, excluding No. 1, which have have goats.

Now there are only two doors closed, the one you picked No. 1, and one that was part of N-1 (999) doors. He then says to you, "Do you want to pick door No. 1 or the other one?" Is it to your advantage to switch your choice?

Suppose there's two doors. You can't decide. The host opens one. There is no goat behind that one. Now you know for certain.

It did not get any more intuitive for me.

Say there are a trillion doors and the host closes 999,999,999,998 of them. Which scenario is more plausible:

1) Your 1-in-a-trillion chance original choice just happened to be correct

2) The omniscient host picked the correct door

Agreed.

But here's the intuition: because the host never opens a door with a car behind it, the fact that he didn't open a particular door suggests that the door in question is the winner. Compare that to the initial door, which was selected blindly.

Why doesn't it also suggest that my door is now more likely to be the winner?

The host isn't allowed to open your door, regardless of what's behind it. That's part of the game

The most succinct description I've ever found of why you switch was so good I made a note of it, though sadly not the provenance:

"If you stay with the door you picked initially you succeed if the initial door has a car, which has a chance of 1/3. If your strategy is to switch then you succeed if your initial pick is a goat, which has a chance of 2/3."

How about this:

You picked a door. It has either a goat or a car behind it, but the goat is more likely. In which case, the remaining doors have a goat and a car behind them. And the host has just kindly revealed which one has the goat.

Let me try. Say you pick the prize door initially. Obviously, if you switch you will now have a non-prize door.

Say you pick one of the two non-prize doors initially. The host now opens the other non-prize door, leaving only the prize door. If you switch you get the prize door.

So switching always changes the prize door to a non-prize door and a non-prize door to the prize door. You start on the prize door 1/3 of the time and a non-prize door 2/3, so you should always switch.

Actually, this makes sense! Thanks.

In teaching a bunch of discrete math classes over the years, I've very rarely found someone for whom the "imagine it's 1000 doors" version helps at all. The only way that gives any insight is if you've already internalized the correct answer. It makes the numbers more extreme; it does nothing to make them make intuitive sense.

The version that helped you is the version that helps nearly everyone in my experience.

I think the 1000 doors version does help, if you clearly explain the point that "now, if you switch, the only way you lose is if you had guessed the right door out of the 1000!"

You can pick one door, or all the other doors. What do you pick?

I've thought about this problem a long time, and the only thing that allowed me to wrap my head around it was to draw it out on a small piece of paper.

Draw three rows of three rectangles each. Now draw a star (representing the prize) in row one, box one, another star in row two box two, and finally a star in row three box three. This depicts all possible "game states". Now we will play the game three times, but in each case you will always choose the first box in each row as your first chosen door. Now the host will open one of the other doors. In the first row, you have two choices to open, so pick one and draw a circle in it. This door is open revealing nothing. In the second and third rows, you must not open door with the prize, so you choose the empty box in each row.

Now go through and tally up how many times you would have won if you had not switched doors and stayed with your original guess (1/3). The tally how many times you would have won had you switched (2/3).

The explanation that clicked for me was doing this with a deck of cards. If I put them all face down and tell you to pick the ace of spades. You choose your card randomly and I flip over all the others except one. So now the ace is either the one you picked or the one card I didn't flip over.

Clearly, you should switch.

The thing that made it click for me is to think about having multiple trials.

Only one of the three doors has the car. So you have a 1/3 chance of correctly picking it and 2/3 of the time you'll pick a wrong door.

Monty opens a door that he knows doesn't have the car. Since 2/3 of the time, you've picked a goat, knowing where the other goat is means switching will guarantee you get the car in those 2/3 of the trials in which you picked a goat initially.

So if you always stay with your initial pick, you keep a 1/3 success rate over n trials. By always switching, you get a 2/3 success rate over n trials.

It didn't get any more intuitive for me like that. If anything, less.

When it clicked for me is when I thought of it as being 1/3 probability the car is behind your door, and 2/3 it isn't. That doesn't change whether they show you one of the other two doors or not, so switching to the 2/3 option is better.

You are only a small step away from "assume a spherical cow" or something involving Hilbert's Hotel.

Which part of three (doors) are you having snags with?

This one is uniquely impressive in that often when first described to someone they will be all but certain it doesn't matter.

Many I told were just about ready to argue their view to death and asserted the switch your choice conclusion is wrong despite them having no formal learning on the subject.

This and the birthday problem effectively illustrate how intuitively bad we are at probability and statistics in general.

Properly applying Bayesian inference gives you the correct answer.

Also the answer becomes intuitive once this scales to more than 3 doors (someone below already mentioned).

The best explanation on the solution for this problem is from Numberphile.

https://www.youtube.com/watch?v=4Lb-6rxZxx0

One of the things that interests me about this paradox is that when you're left with just two doors, the odds of either of them being the one with the car is 50%, so what does it matter if the one you happened to pick was picked before the host opened some other doors? How could that possibly affect the probability? It seems like choosing to switch would be admitting that the host's action has some kind of reverse-causality.

The other thing I find interesting about this puzzle is wondering whether the people who find the arguments in favor of switching convincing are actually reasoning better than I am. If so, how are they doing this? What is it that makes those (allegedly superior) arguments convincing to them?

The thing that kind of convinced me that the pro-switch arguments must be correct is that people have run simulations of this problem on computers and the switching strategy turns out to win in line with the pro-switch argument probabilities. But the tricky thing about probabilities and randomness is that such successes could be "just a matter of luck", as it were. Maybe they were wrong, but just got lucky. It's improbable, but still possible.

Only mathematical proof independent of experimentation should be rock-solid correct. But mathematical proof seems to be a matter of being convincing enough to mathematicians, and no so independent after all, and that seems disappointing somehow. There's a feeling that mathematical proof should be above mere argumentative skill and the audience's susceptibility to being convinced.

> One of the things that interests me about this paradox is that when you're left with just two doors, the odds of either of them being the one with the car is 50%.

Well, no, that's the point of this, it's not 50%. Before you pick there is a 1/3 chance of the car being behind any of the doors, so there is a 2/3 chance of it cumulatively being behind the other two. But there is only one car, so the set of doors you didn't pick must include at least one non-car. The host knows the location of the car so can always open a non-car door from this set. But still as a set they have a 2/3 chance of containing the car. By swapping you can effectively think of it as choosing both other doors. An easy way to get your head around this is increase the number of doors, say to 100. You pick 1 of those 100, the host opens 98 other doors, do you swap?

Alternate reality rules of the game are: nobody speaks, you pick any two doors you like, and you win the prize behind both of them. The odds of wining this game is obviously 2/3. Monty will gesture at you to point to a door. You point at the one you don't like. Monty will then show you one of your prizes behind one of your two chosen doors. Monty will then gesture for you to point to a door. You point at the remaining one you liked. Monty then gives you your second prize behind that door.

Notice that all of the externally visible actions are identical to the original game.

>You pick a door, say No. 1, and the host, who knows what's behind the doors...

I don't get this. Why does't the player insist that they open door no 1. Or is it part of the game that the host will open a different door other than the one the player picked...

If the host knows what is behind the doors, and you are only given two chances, and the host can open a different door from what you picked, isn't your chance of winning zero?

I don't get this...

The host's job is to create suspense by opening a door and offering a chance to switch. If they opened the door you picked first, there would be no suspense, and if they didn't know what was behind the door, they could show you that you lose, which also removes suspense.

Turns out the host just ends up giving you more information that you can use to make a better decision.

Can you please explain this..

>If the host knows what is behind the doors, and you are only given two chances, and the host can open a different door from what you picked, isn't your chance of winning zero?

When we discussed this in school I quickly wrote a program that simulated it ...

No need to look at fancy paradoxes, just think about the following.

What does it mean that tossing a fair coin has a 50 % probability of showing heads?

If you think you know the answer, you are probably wrong.

EDIT: Instead of just voting this down, try to give an answer. If you think it is easy, you have not thought about it careful enough.

The frequentist answer is that you could get the proportion of tosses that come up heads arbitrarily close to 50% in a large enough sample.

The Bayesian answer is that given all the evidence available to me I would be willing to bet $1 to win $1 if the next toss comes up heads. (Since I work in finance I'll add that this assumes I'm risk neutral, so losing $1 is exactly as bad for me as winning $1 is good.)

Except for the risk-neutrality detail this is all Probability 101, right? Or are you thinking of something else?

That the relative frequency converges to 50 % is obviously not true. You would have to be exceptionally lucky, but you could get heads, heads, tails repeated for ever and therefore the relative frequency of heads would fluctuate increasingly tiny amounts around 66.(6) %. This of course has probability zero, but it is not impossible. And there are of course many other sequences of outcomes for which the relative frequency does not converge to 50 %. So at best you could say that the relative frequency converges to 50 % with high probability, but now you have a circular definition because you make use of probabilities while defining probabilities.

The Bayesian view is problematic for several reasons. Why do I need someone with believes about the coin, we are talking about intrinsic properties of tossing a coin after all. And if that is not enough, we also throw some betting in. Tossing a coin does certainly not depends on the invention of money and gambling, at least ignoring that coins are usually money. Last but not least you have to explain where your believe about a 50 % probability for heads comes from and how it is justified. I could certainly believe that the probability for heads is 25 %, that would not be a good believe.

So, what's the right answer?

If I knew, I wouldn't have asked. And as far as I can tell nobody knows or at least there is no commonly accepted answer.

I hope to have convinced with my previous answer that the mathematical definition of probability is key to resolving your paradox here. Mathematically (using the concept of infinity) and also physically (using the concept of large numbers) there is a simple and commonly accepted answer to your question.

Since that definition is very basic, it is hard to find a direct answer to your question.

You might find it interesting that in physics when we deal with large numbers that are not infinity it is usually good enough to know that a certain event is pretty damn unlikely (and it is essential to corroborate this with numbers as in "this happens once in 10^100 years"). In physics, we rarely have 100 % certainty. Logical consistency and a certainty that is large enough are usually key to success (i.e. growth of knowledge) in natural sciences.

I guess another way of stating the problem is this: say you toss a coin and get heads the first 1000 times. Now you are likely to believe that the coin is biased. But that cannot be proved. It may be that you have simply not tossed the coin enough times to perceive its fairness. Maybe after the 1000th toss you start to get enough tails such that after the millionth toss, it is not at all clear that there is any bias.

So practically we can make a judgement as to whether the coin is biased on not based on how many tosses we think is sufficient, but theoretically it is impossible to distinguish a biased coin from a fair one if we toss to infinity

Yes, that is exactly what I had in mind. Probabilities are weired in the way that they say what will happen but then still leave open the possibility that this will not happen at all. You toss a coin a billion times, you will get about 500 million heads. Well, or you don't and there are exactly zero heads.

> theoretically it is impossible to distinguish a biased coin from a fair one if we toss to infinity

correction: if you replace infinity by any large finite number your statement is correct.

>That the relative frequency converges to 50 % is obviously not true. You would have to be exceptionally lucky, but you could get heads, heads, tails repeated for ever

This would violate the law of large numbers. You may end up with the sequence HHT 10million times, but the chances that you continue to get that sequence for another 10million times is all but zero, and then gets even smaller as you add another 10million trials. As the number of trials approaches infinity, you will arrive at 50%.

No, the law of large numbers does not assert that all sequences of outcomes converge, only that this happens almost surely. Or look at it the other way round, what mechanism would prevent heads, heads, tails repeated forever? I can certainly get heads, heads, tails on the first three tosses. After that I start over, three more tosses all independent of what just happened, again a 12.5 % chance for heads, heads, tails. Why could this not continue forever?

I'm pretty sure your confusion about probability stems from you not understanding the mathematical concept of a "limit". Your sequence HHT has a 12.5% chance of happening, but so does the sequence TTH and as the number of trials approaches infinity, you will get similar counts of the two because they have equivalent probabilities.

No, I will get similar counts with high probability but not surely. There are sequences of probability zero that do not converge. Think about it this way. Every coin toss in a sequence, finite or infinite, on its own can surly turn out to be heads, can't it? And all tosses are independent, aren't they? So why can't all tosses turn out heads? Unless you have a convincing argument why some tosses have to yield tails eventually, you have to deal with the fact that not all sequences of tosses converge to the expected probability.

> Every coin toss in a sequence, finite or infinite, on its own can surly turn out to be heads, can't it?

I would agree if the sentence contained just the word "finite". The "or infinite" is where you are thinking too intuitively, and not mathematically correct anymore. The difference between "finite" and "infinite" is precisely the solution to this paradox in your mind.

I hope to be able to point out that it is easy to correct for that with just a bit of structured (but maybe non-intuitive) thinking.

One of the simplest and (I would argue most complete) definitions of "a probability of 0.5 for both test outcomes A and B" is that, given an infinitely large sample size, half of the samples show result A, while the other half shows result B. Think about this for a second, and I recommend to also use this opportunity to appreciate again that half of infinity is still infinity.

This relates the mathematical concept of infinity to the definition of probability.

With this definition, it probably feels like I so far just reworded your question. That may not be satisfying. So, I would like to encourage you to imagine that you have superpowers and can actually perform an infinite number of tests. You do that on a sunny day and observe that all test outcomes were the same: A. You call it a day and you can conclude (using the mathematical definition from above) in your diary of days-with-superpowers: "Today I have empirically determined that the test shows outcome A with a probability of 1". You might smile and add "Peter said that outcome A has a probability of 0.5, but I have proven him wrong".

In other words: if you do an infinite number of tests, the normalized distribution of test results precisely is the probability distribution of test results.

I think we have learned by now that the concepts of infinity and probability are deeply related and can, by definition, be used to explain each other. That might still not be satisfying. So, I would like to focus on the "finite" case for a bit.

Imagine you don't have super powers anymore, but you're pretty resilient and motivated and you want to do the experiment to (in)validate Peter's claim: "The probability for both, outcome A and B, is 0.5 each!".

After 1.333.337 tests you have seen 1.333.337 test results showing A. You're tired from all the testing and you complain (correctly!): "it is now really pretty unlikely that Peter is right! I am pretty damn sure that he is wrong! How long do I still need to do this to be absolutely sure?" -- and then a voice from the darknet reminds you: "for being absolutely sure, that is, for finding an answer that is correct with a probability of 1, you need to have super powers and make an infinite number of tests -- sorry dude, you can't do that, ever, because it's inconvenient, takes infinitely long and such -- so I need to disappoint you, you will never be sure, but maybe just enjoy your life as much as you can".

Infinity usually does not allow for actual intuitive thinking. But there are a few really simple mathematical rules around infinity and convergence that make it actually pretty simple and again intuitive to deal with the concept.

I appreciate your attempt but you did not convince me the slightest bit. Let's take the 1,333,337 coin tosses all heads. This result has no bearing on the probability of the coin at all. It may make you strongly doubt that the coin is indeed a fair coin but - and that is the point I am trying to get at - there is nothing that prevents a fair coin from coming up heads 1,333,337 times in a row. Whatever your experiment shows, it could always be a statistical fluke.

And the infinite case does not changes much, at least not in a way obvious to me. Back with those super powers I toss the coin infinitely often and get heads 50 % of the time. That was fun, let me do that again tomorrow. Well, again heads 50 % of the time. This is the way it goes for a long time but then something strange happens, one day all tosses come up heads. The very next day everything is back to normal. What is now the probability of heads, we got two different answers for your way of defining he probability? And all it took was an extreme statistical outlier on single day.

Not OP but I think you still don't grasp the concept of infinity. Terms like "the very next day" or any other segmentation don't apply.

> What is now the probability of heads, we got two different answers for your way of defining he probability? And all it took was an extreme statistical outlier on single day.

The point of probability is that performing an experiment an infinite number of times guarantees that every outcome happens with a proportion that exactly equals its probability (for a formalization of what that even means, look at measure theory). If you get different proportions on different days, you have different probabilities. That means, you weren't performing the same experiment.

That's why I say you cant prove a probabilist/statistician wrong about probabilities of an outcome in any finite amount of time.... Its always good to be in such a business except for they cant be proven right either (in finite time)

I agree, there are problems and non-intuitive aspects to both approaches to probability. Otherwise there wouldn't be two approaches.

I think your objection to frequentism unfairly conflates the idealized world with the physical world. Only in the idealized world can you assert a priori that a coin has 50% probability of coming up heads. In that world you can toss the coin infinite times and it almost surely comes up heads close to 50% of the time.

In the physical world you merely have a model stating that heads will come up 50% of the time. In this world you could toss the coin millions of times and have it come up heads 66% of the time, but all you've done is provide really strong evidence that your model is wrong.

Also, you left out 2 arguments against frequentism: it allows inconsistent beliefs, and in practice it has allowed bad approaches in scientific papers.

As for the Bayseian view, being non-intuitive isn't the same as problematic. The Twins Paradox https://en.wikipedia.org/wiki/Twin_paradox violates our intuitive understanding of how time works, but that's because our intuition is wrong in some conditions. You thought when I said the coin had 50% chance of coming up heads that I was making a statement about the coin, but really I was making a subjective statement about how I would bet. If you believe it's 25% then there's a clear way for us to resolve our different beliefs.

I think the interesting part is not that the relative frequency might converge to something other than 50 % under non-ideal conditions, but that it might not converge at all, admittedly only very, very rarely. And this seems to force you into an infinite regress.

50 % probability for heads means that if you toss the coin infinitely often the relative frequency will converge to 50 %. But not quite, in very rare cases it won't. So you have to toss a coin infinitely often an infinite number of times and then you will see that all but a tiny fraction of the experiments indeed show the relative frequency converge to 50 %.

But now you have to quantify that this tiny fraction is something of probability zero. And even worse, it is still possible that none of your repeated experiments showed convergence, you seem right back where you started.

I would love to know to what you are referring with the inconsistent beliefs.

I would not say that Bayesian view is non-intuitive, I would say it fails to account for important things. A priori probabilities have to be rooted somewhere. Where does you beliefe in a 50 % probability for heads come from? Because you have previously observed the relative frequencies of coin tosses? Because you made some theoretical observations about symmetries?

There must be, at least so it seems to me, something about the probabilities associated with coin tosses that is independent of any individual, otherwise it would become rather difficult to explain how different individuals would arrive at similar probabilities independent of each other. So banning probabilities into the realm of beliefs does not cut it in my opinion.

Well, you can build a physical model of a coin, and all forces acting on it during toss/fall, and show that in an ideal scenario (perfect coin/perfect landing surface), there would be roughly half of the initial conditions leading to heads, and roughly half leading to tails.

If you want to go down this road, then we will have to switch to a nuclear decay based coin or something like that. In the case of a coin toss there never was any real randomness, as you say we were just ignorant of the initial conditions. Given a distribution over the possible initial conditions, we can determine the probabilities for heads and tails. The possibly huge number of degrees of freedom and deterministic chaos will of course make this an unpleasant exercise.

Can we base our belief about coin toss probability on the belief that about half of the initial conditions lead to heads and half to tails, without verifying it?

> I would love to know to what you are referring with the inconsistent beliefs.

You could build frequentist models for the odds of 0 to 1 inches of rain falling in Cleveland tomorrow, 1 to 2 inches, and 0 to 2 inches, and the odds of 0 to 1 plus 1 to 2 don't need to add up to 0 to 2 inches. You can justify all 3 models, but they are inconsistent. Bayesian models don't allow that. I read about this in Aaron Brown's "Red-Blooded Risk". Here's an online source asserting "frequentists can have two different unbiased estimators under the same likelihood functions." https://chenghanyu.wordpress.com/2014/03/26/the-strengths-an...

Here's one saying that Bayesian models are self-consistent: http://bactra.org/notebooks/bayesian-consistency.html

Here's Eliezer Yudkowsky giving examples of my other point, that frequentist models can encourage sloppy behavior from scientists: http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...

> So you have to toss a coin infinitely often an infinite number of times

This is a core piece of your confusion. Infinity isn't a number; you can't multiply it by another number and get a meaningful result. You are saying you could run an infinite number of tests on Monday and then run an infinite number of tests on Tuesday and so on, but the concept of infinity holds that you did as many trials on Monday as you did on Monday and Tuesday combined. (Inf * 2 = Inf).

It means that there is a 100% chance that one side of the coin has a head on it.

Well, that is already in the setup, there is also a 100 % chance that the other side is tails. I don't think that observation will turn out to be very useful.

That we're modeling the outcome of a coin toss as a sample space set containing at least two event elements, one of which we call "heads," and that we have a probability measure which assigns 0.5 to the subset consisting of only the "heads" event.

You are presupposing an understanding of probability, aren't you? What does it mean that a probability measure assigns 0.5 to heads?

If it's a fair coin, then by definition there's a 50% probability of showing heads.

Yes, but what are the consequences of this definition? What does a probability of 50 % for heads imply about the coin? Or what is the difference to an unfair coin with a 40 % probability for heads and a 60 % probability for tails?

I think you're answering your own question.

I don't think so, I personally can not explain to you the difference between a 50/50 and a 40/60 coin in a way that would satisfy me.

Are you interested in joining a weekly poker game? I'd love to have you at mine.

The probability has no implications about the coin. It has implications about how you make decisions concerning the flipping of coins.

It almost sounds to me as if he is pushing into gambler's fallacy space. Like if a (fair) roulette wheel hasn't hit 7 in 200 spins, then 7 is somehow due.

No, his arguments is exactly opposite to the gambler's fallacy. If the gambler's fallacy were not a fallacy, then there would be no issue. But because we know it is a fallacy, then the fact that we have 1000 heads so far tells us (in theory) nothing about the fairness of the coin. But in that case, what does a probability of 50% actually mean?

> the fact that we have 1000 heads so far tells us (in theory) nothing about the fairness of the coin

Really? If I flip a coin 1000 times and always get heads, how much would you be willing to bet me that the next flip gets tails?

So after how many consecutive heads will you decide that the coin is biased? 10? 100? 1000? And what is that number based on?

In any case, if you choose any number X, then what you are really saying is that a fair coin cannot produce X heads in a row. In that case, consider this thought experiment: lets assume we know a coin is fair, then by your logic, as soon as we have X-1 heads in a row, we already know the outcome of the Xth toss: it should be a tail. But this is a contradiction, because if the coin is really fair, then the probability of a tail on any toss should always be 50%, irrespective of what has happened before

You can play this game for every statement.

What does it mean that an elephant is bigger then a dog?

If you think you know the answer, you are probably wrong.

That one is easy. First let's clarify what exactly we mean with bigger. Let's say we mean larger in volume. Now we pick a procedure for determining the volume of animals, say we submerge them briefly in a large water tank and mark the resulting water level. That animal that lead to the highest water level is the bigger one. Any objections?

But similar to your objection about the Bayesian view of probability requiring money and betting to be defined, you've now defined the idea of size to be dependent on giant tanks of water

    Any objections?

The same objections you have regarding the fair coin.

You say you submerge the animals in a tank of water. How does that work? You have this giant tank, put in the animal and one molecule of water? Just like a single toin coss, a single molecule of water will not tell you much.

You need hundreds of billions of water molecules?

Ok. And how do you know these will behave in the expected way?

Because you have a model in your head how molecules behave? Well, I have a model how fair coins behave.

Because you saw different sized objects result in different water levels before? I saw fair and unfair coins result in different head:tail distributions before.

By far the most unintuitive paradox for me personally is the one presented here: https://youtu.be/go3xtDdsNQM?t=3m27s

"Mr. Jones has 2 children. What is the probability he has a girl if he has a boy born on Tuesday?" Somehow knowing the day of the week the boy was born changes the result. It's completely bizarre.

Is there a more rigorous explanation of why they count the probability space how they do? Watching that video I feel like the ordering of the kids and the striking of one of the "b2b2" entries seems wrong to me. If we care which kid was first... which doesn't seem to matter... then the first b2b2 and the second b2b2 seem like they're different and shouldn't "cancel."

Then again... it took me multiple explanations to understand the Monty Hall problem.

Yeah, it definitely seems wrong to me. B2B2 has twice the probability of any other event listed in that table.

edit: I guess it's more nuanced than that. The explanation and interpretation sections on this blog post [1] and on wikipedia talk about the controversy.

[1]: https://jakubmarian.com/the-day-of-the-week-boy-or-girl-para...

[2]: https://en.wikipedia.org/wiki/Boy_or_Girl_paradox

Even if that's the case it still doesn't make sense. That changes the probability to 50% girl instead of 52%. But it should be 2/3rds.

dammit, you're right.

This B2/B2 elimination also immediately jumped out to me as being wrong.

But a Youtube comment wrote: > "But the thing is, I ran a few tests through a big randomized sample set, and... he's right. It's super weir … the second boy-girl problem had ~51.9% change of containing a girl. Keep in mind this was about 100 million randomized samples too."

I can't wrap my head around it.

So I think intuitively, something that might make sense to you is that the reason that BB is more common with the restriction that a boy must be on Tuesday is that having 2 boys increases the probability that you will have a boy that was born on Tuesday.

Here's another way to think about it. It's twice as likely to have one girl and one boy (either BG or GB) compared to having 2 boys. Thus, the version of the paradox where you're given that the parent has a boy results in a 2/3 chance of the other child being a girl.

However, the more unlikely it is that any given boy satisfies the condition (in this case the condition would be being born on a Tuesday), the more likely the BB case becomes compared to BG or GB.

More concretely, if only 1/n of the boys satisfies some condition, you would be left with only 1/n of BG or GB, while you would be left with 2/n - 1/(n^2) of BB. In this case, let the population be all parents with at least one boy (this consists 1/3 of BB, 1/3 of BG, and 1/3 of GB). Letting n = 7, (BG union GB) represents (1/7)(2/3)=2/21 of the population, while BB represents (2/7 - 1/49)(1/3) = 13/147 of the population.

Now, (2/21)/(2/21+13/147) = 14/27, our desired result.

I hope that's a more intuitive way of thinking about it. The important part is realizing the differences between knowing that the parent has a boy vs knowing that the parent has a boy born on Tuesday.

>while you would be left with 2/n - 1/(n^2) of BB

For those like me who didn't immediately understand where this came from:

    P(atleast one of two boys satisfies the condition)
    = 1 - P(neither of two boys satisfies the condition)
    = 1 - [P(a boy doesn't satisfy the condition)]^2
    = 1 - [1 - P(a boy satisfies the condition)]^2
    = 1 - [1 - 1/n]^2
    = 2/n - 1/n^2

>having 2 boys increases the probability that you will have a boy that was born on Tuesday.

I guess. But the fact that it happened to be Tuesday isn't really important. The person could just as easily have had a kid on Wednesday. And all the logic would be the same. And the kid has to be born on some day of the week. How does finding out what day it was give us any additional information about the other child? It's completely independent!

So that's what confused me to begin with as well.

I think the main thing here depends on your interpretation of how the parent is chosen.

Let's say the question is the same, but we relax the requirement that the boy is born on Tuesday. Do you think the probability that the other child is a girl is 2/3 or 1/2?

I think there is a whole class of statistical "strangeness" with using p values for hypothesis testing. For instance, p = 0.05 means that we have ~30% chance that our hypothesis is a false positive [1], which is far from what intuition tells us.

[1] http://www.nature.com/news/scientific-method-statistical-err...

"our hypothesis is a false positive"

I wonder what you mean by this. In what sense can a hypothesis be a false positive?

I mean false positive that null hypothesis is rejected.

If you throw a six sided dice two times, there's 1/6*1/6=~3% chance to hit six both times. But if you throw one six, there's now ~17% change to hit six again ...

Here is another seemingly basic question that leads down the rabbit hole: What does it mean to say two things are the same?

my favorite data science paradox is the "curse of dimensionality"

https://en.wikipedia.org/wiki/Curse_of_dimensionality

Reply on news.ycombinator.com