What are the most important statistical ideas of the past 50 years? [pdf]
stat.columbia.edu
stat.columbia.edu
Link to the free HTML version
Submissions like this give me hope, that if I just read more, I'll get there, but nothing listed here feels like it's usable in a "napkin math" scenario. I just bought two of Gerd Gigerenzer's latest books, and I've already obtained some of David Sklansky's more introductory work on gambling, but I'm concerned my focus is either too theoretical or too narrowly on gambling (though I think there's a lot to learn in how a gambler assesses a bet, and how it applies to other situations).
I guess if it were easy, we'd all be doing it...
Suppose I want to make a decision about whether to hedge for a market crash right now. Statistics can tell me the likelihood of a crash, and how bad. But if the market crashes, and very badly, how might that affect my life? To make a good decision I would need to think of all the things that come with a market crash (job loss, savings loss). This is not statistics.
I could again use statistics to say what is the chance I lose my job given a market crash (say 70%). But then I would need to estimate the impact on my life should I lose my job (Stress, etc). This is not statistics. But it should very well factor into my ability to do back of the napkin math on whether I should hedge or not.
In particular:
> But if the market crashes, and very badly, how might that affect my life? To make a good decision I would need to think of all the things that come with a market crash (job loss, savings loss). This is not statistics.
This is all statistics, not just the part where you're forecasting likelihood of the market crashing. The reason is because making decisions about the future under the constraints of uncertainty implicitly involves a forecast. When you decide how to diversify your personal investment portfolio, how much to allocate to your Roth versus traditional IRA or 401k, etc, you are making forecasts about which allocation will provide you with a more favorable outcome.
Stated more concisely: there is no rational reason to use statistics for forecasting market events but not for deciding what to do in the event specific market events occur.
Statistics cannot tell you any such thing.
What is a likelihood, but a statistic?
If there is any method to determine a statistic, it seems reasonable to me to say that that method involved statistics.
(Now, of course, except for possibly where quantum randomness is relevant, which might be quite often, I'm fairly confident that the only probabilities are subjective or relative to some set of assumptions, or something along those lines, because the future "already exists". But, given some fixed priors and some fixed evidence, there should in principle be a well defined probability of such a crash. So, insofar as peoples priors match up, there should, in principle, be a common well defined probability given "the information which is publicly available", or also, given whatever other set of evidence.)
Of course, that doesn't mean it is computationally tractable to compute such a probability.
:-)
How do you test this model?
It is easy to find things that fit one of the previous crashes.
Given that there is only one realization of history, the data we have is consistent with any model that puts a non-zero probability on a crash.
So, I'm not sure what you mean by "test this model".
You can refine your model-of/beliefs-about the world, by continuing to look at the world and make observations.
And obviously your beliefs should include a non-zero probability of a crash. That follows from non-dogmatism/Cromwell's rule.
And yeah, there is only one, (or, either that, or at least we can only observe one, which is practically the same thing) "realization of history". This doesn't produce any difficulty, because probability isn't defined by the proportion of trials in which the event occurred.
Probability is about degree of belief (or, belief and/or caring).
edit: I suppose you can also evaluate how calibrated your beliefs have been, which is kind of like testing a model.
Not at all.
Probability is a countably additive, normalized measure over a sigma algebra of sets.
> This doesn't produce any difficulty, because probability isn't defined by the proportion of trials in which the event occurred.
You misunderstand the point.
Let's say you provide me a distribution of crash probabilities for every trading day for the next three months.
We all ought to know that P(event) = 0 does not mean event is impossible., Therefore, P(event) = 1 does not mean "not event" is not impossible.
What would allow one to state that your model is consistent/not consistent with the one observed history of events over the three months, regardless of whether there is a crash or not?
You have to come up with this criterion before observing the history.
Ok.
I would think that, if we have a continuous distribution, then the score should be the probability density of what is observed?
If you say beforehand “I think x will happen”, and I respond “I assign probability 1 that x will not happen”, and then x happens, then I’ve really messed up big time. I’ve messed up to a degree that should never happen.
(And, only countably many events can be described using finite descriptions, and a positive probability could, in principle, be assigned to each, while having the total probability still be 1, so that nothing that can possibly be specified happens while being assigned a probability of 0. Though this isn’t really computable..)
As a more practical thing, if I assign probability 0 to an event which you could describe in a few sentences in under 5 lines (regardless of whether you actually have described it), and it happens, then I’ve really messed up quite terribly, and this should never happen (outside of just, because I made an arithmetic error or something.)
I think this conversation has reached an impasse.
"the measured value is in the cantor set" isn't a thing that we can observe to have happened.
("the value, when rounded to the finite amount of precision that our measurement has, is in the cantor set" is something that would have positive probability, under the uniform distribution over the interval, and therefore something I shouldn't assign a probability of 0.)
Economics is not Statistics (and definitely not statistics).
Most of the discipline focuses on testing models and making inferences on observational data. The techniques for dealing with that sort of data, of course, build on Statistics, but their nature is different enough that there is Econometrics.
A large part of economics is not empirical at all -- despite the fact that people get Nobel prizes pretending this not to be the case.
Even in the context of experimental economics, since the behavior of the observed vary depending on the mode of observation, the contexts in which the most straightforward Statistical methods designed to apply to engineering/chemistry/biology experiment type situations are not directly applicable (although it is great when they agree with the fancier methods).
I'm not sure which parts of the field or which prize winners you are talking about. To be clear: you think economics is _not actually empirical_, but people are awarded Nobel Prizes for _pretending that it is_? That's a little odd. Let me know if that's not what you meant.
When you look at this list:
https://en.wikipedia.org/wiki/List_of_Nobel_Memorial_Prize_l...
Who satisfies that condition, in your mind? Who is getting the prize on the basis of pretending that economics is empirical?
That is a misrepresentation of what I said.
To be clear, I think what I said:
>>A large part of economics is not empirical at all
E.g., as an example, Kahneman's Nobel is solely a product of taking an axiomatic theory and designing experiments where regular people who are actually not being paid according to their performance are gently prodded into violating the axioms in weird settings. It is attractive to people who want to claim that clearly the plebes cannot be allowed to choose for themselves as they are not "rational".
The only meaning of "rational" in Economics is that individuals choose the best alternative according to their preferences among a constrained set of alternatives. Here an "alternative" or "bundle" is a point in the entire commodity space.
The only test of this is consistency with GARP: A choice is not rational if a feasible and more preferred alternative exists.
A useful parallel can be drawn with situations involving measurements and data, since data often have the same feature of telling us something but not everything. This is what I believe makes statistics so useful for science.
Jake Vanderplas's presentation https://speakerdeck.com/jakevdp/statistics-for-hackers can give you some concrete ideas of how far you can get with just a random number generator.
On a related note, one of the last times I mentioned Hubbard here, another book came up in the surrounding discussion, which looks really good as well. Guesstimation: Solving the World's Problems on the Back of a Cocktail Napkin[2] - I bought a copy but haven't had time to read it yet. Maybe somebody who is familiar will chime in with their thoughts?
[1]: https://www.amazon.com/How-Measure-Anything-Intangibles-Busi...
[2]: https://www.amazon.com/gp/product/0691129495/ref=ppx_yo_dt_b...
- how to measure anything; Hubbard actually comes through on the promise of the title - after finishing the book you will truly feel that the scope of what you can measure is massive. He does this by a change in the definition of what it means to measure something, but you realize his definition is more correct than the everyday intuitive one.
- value of information; Hubbard gives a good introduction to the VOI concept in economics, which basically lets you put a price on any measurement or information and prioritize what to measure
- motivation for 'back of the napkin' calcs; through his broad experience he has seen how a lot of the most important things that affect a business go unmeasured, and how his approach to 'measuring anything' can empower people to really measure what matters.
Reading this book provided one half of what I have been searching for for a long time - a framework for thinking about data science activities which is not based on hype, fundamentally correct and still intuitive and practical.
Stats is a tool like any other tool. Boil your question down to the fundamentals (first principles) and maybe stats is one of the tools you decide to use to solve part of it, where appropriate.
Most questions involving data can be answered with something as basic as a plot.
"mathematical modelling of nature"
Basic reasoning, tinkering, plotting data and playing around with data in Excel or Python is usually sufficient to answer most questions, yet many will try to overcomplicate the issue with complex math or stats. Perhaps in an attempt to impress their peers, or perhaps because they've just come out of years of university that taught them to think theoretically.
I'm reminded of that meme going around.
70 IQ: plot data
100 IQ: support vector machines! deep Q-nets!
130 IQ: plot data
It's usually simpler than it sounds, and helped me put together a lot of quick decision models, sales forecasts, etc.
https://en.m.wikipedia.org/wiki/Maximum_entropy_probability_...
Being comfortable around basic probability distributions is probably the main prerequisite I suggest. If you can understand the uses and analyses of Bernoulli, Binomial, Poisson, and normal RVs, you should be at a point where you can try to model real life phenomena. A surprising amount of life can be modeled as multiple weighted coin flips.
What kind of decisions do you want to make? If you offer that up, others will be able to provide guidance to frameworks used to make those decisions.
The field of operations research uses topics from all sorts of fields to try to make better decisions. I would broadly classify modelling into two large categories deterministic (e.g., how to schedule a flight crew, or where to locate a warehouse) or stochastic (using probability models, e.g., how many people to staff in a bank). Of course there is no problem that will be strictly one form or the other. There will be points in a modelling problem where it's useful to apply tools from both categories.
A decision tree might be the simplest form. You map out the different decision points and can generate costs or profits and probabilities for the decision points. You take the one that has the highest expected value (sum of probability of outcome times value of outcome) of profit or lowest expected value of cost.
Sensitivity analysis may be another tool to use in decision making. You don't need to get three decimal accuracy. Come up with some upper and lower bounds and you'll have an idea of how good or how bad the outcome will be.
Determining the probabilities and bounds may be tough, and your decision will only be as good as the data your putting into the framework.
Statistically speaking, it says that regardless of the dimension that data is presented in, a random linear projection to (slightly more than) the intrinsic dimension of the data captures the underlying topology.
Max Planck is quoted, "Science progresses one funeral at a time." In this case, the grand old man of statistics finally died still insisting that it could not be proved that smoking caused cancer, but not before blighting careers of those who were showing it could.
I have heard Pearl make this claim but have never seen his evidence for it.
As a counterexample: Don Rubin is a statistician who has a well-known framework for causal inference who has been at the top of the field for a very long time. Rubin has published widely and very well on causal inference.
Is there good evidence for the topic actually being suppressed by anyone within the statistics profession? There is work on causal inference going back to RA Fisher. If anyone has tried to suppress it, I'm not sure they have been very effective.
Unfortunately there's not enough evidence to show causation...
https://www.amazon.com/Causality-Reasoning-Inference-Judea-P...
Pearl is very careful to give his deceased predecessors their due credit. That their work was suppressed will always be a blot on the leading names in statistics in the past century.
Edit: the word "tail" appears nowhere in the paper, in any context. I'm beyond shocked now.
That would be my suspicion as to why it isn't there.
1. https://en.wikipedia.org/wiki/Extreme_value_theory
2. https://en.wikipedia.org/wiki/Fisher–Tippett–Gnedenko_theore...
3. https://en.wikipedia.org/wiki/Pickands–Balkema–De_Haan_theor...
More broadly, let me leave a Taleb anti-recommendation. His entire shtick is yelling that traditional statisticians have ignored heavy-tailed random variables in their modeling and that he has special insight into the nature of tail risk (perhaps along with a few select other people, like Mandelbrot).
But this is manifestly not the case. In fact, if you go through his Amazon reviews page, you can find him leaving positive reviews several years ago on all the books written by traditional statisticians that he learned about heavy-tailed randomness from!
For a more detailed critique, see Robert Lund, Revenge of the White Swan, The American Statistician Vol. 61, No. 3 (Aug., 2007). Accessible through your favorite Russian website.
If you want a better book on heavy-tailed randomness, I like Didier Sornette's Critical Phenomena in Natural Sciences (subtitled Chaos, Fractals, Selforganization and Disorder: Concepts and Tools).
https://www.researchgate.net/publication/4741329_Revenge_of_...
If you're referring to "All of Statistics" by Wasserman, then there are some significantly easier textbooks to learn statistics from. Depending on the program, "All of Statistics" is a book used by senior undergrads or grad students. Are there more mathematical heavy stats books, yes, but this isn't a casual read for someone who is trying to learn statistics either.
I like "Probability and Statistics for Engineering and the Sciences" by Devore as an intro book. It covers the basics of probability distributions, maximum likelihood and method of moments estimation, ANOVA, and linear regression. Pre-requisite knowledge is probably multivariable calculus, matrix multiplication, determinants, and eigenvalues.
1. Those who want a statistics book to be like a math book: Fewer words and more equations.
2. Those who want a wordy book with little math
Devore's book is in between, which is why I think both camps tend to hate it. It has a decent amount of math, and has quite a bit of text. The text is invaluable: You get information about common rules of thumb. You get insights on why the technique works. Etc.
And the examples/problems are great. So many of them are from real papers/books. You're not working on some contrived example, but on real world problems.