Hacker News Reader

> It seems clear to me that Popperian falsification is indeed the only way to separate the wheat from the chaff in theoryspace, however, generating theories generally requires inductive reasoning.

A report analyzed the entries to Nature one year and found that very, very few of the papers actually met Popper's criteria for falsifiable hypotheses, and in fact most of the papers started out with an exploratory aim and documented their findings. This means that to adopt the Popperian idea that only falsifiable claims are science means we must reject good science. One must also consider what opportunities are being missed if we were to force good science to adopt strictly falsifiable hypotheses before research commences - this would mean that every exploratory paper would need to be redefined (or rather, firstly defined) as a falsifiable hypothesis... in an attempt to please Popperians.

It is also a stretch to say that science must be empirical or carried out in a particular way to which falsifiability is congenial. There are good arguments that even philosophy[1] or mathematics can be considered sciences, and indeed they were (see Wissenschaft in Kant and Hegel for instance).

Furthermore, it's been argued that Popper's theory of falsification actually includes bad science (pseudoscience). From SEP[0]:

>Strictly speaking, his criterion excludes the possibility that there can be a pseudoscientific claim that is refutable. According to Larry Laudan (1983, 121), it “has the untoward consequence of countenancing as ‘scientific’ every crank claim which makes ascertainably false assertions”. Astrology, rightly taken by Popper as an unusually clear example of a pseudoscience, has in fact been tested and thoroughly refuted (Culver and Ianna 1988; Carlson 1985). Similarly, the major threats to the scientific status of psychoanalysis, another of his major targets, do not come from claims that it is untestable but from claims that it has been tested and failed the tests.

[0] https://plato.stanford.edu/entries/pseudo-science/#KarPop

[1] https://web.archive.org/web/20170621073301/http://www.philos...

Great points. It's one thing to say that a scientific theory should be falsifiable (as a necessary but not sufficient condition), but the Popperians lose me when they claim either (1) that the only valid scientific study is an attempt at falsification, (2) that corroboration (or in Popperian terms, a failed attempt at falsification) has no significance. Both of these tropes come up in this "response", where Ionides et al. use them to throw some pretty serious shade at the ASA:

> This approach to the logic of scientific progress, that data can serve to falsify scientific hypotheses but not to demonstrate their truth, was developed by Popper (1959) and has broad acceptance within the scientific community. In the words of Popper (1963), “It is easy to obtain confirmations, or verifications, for nearly every theory,” while, “Every genuine test of a theory is an attempt to falsify it, or to refute it. Testability is falsifiability.” The ASA’s statement appears to be contradicting the scientific method described by Einstein and Popper.

However, (1) completely ignores discovery science (something I have ranted about here previously), and implies fully accepting Popper's claim to have solved the centuries-old "problem of induction" [1]. Meanwhile, (2) seems to pop up pretty frequently even though I'm not sure Popper himself would have agreed, considering [2]:

> It is easy, [Popper] argues, to obtain evidence in favour of virtually any theory, and he consequently holds that such ‘corroboration’, as he terms it, should count scientifically only if it is the positive result of a genuinely ‘risky’ prediction

Ionides et al. conveniently left out the second half of that context. Hmm, if only there were some sort of way to quantify that "riskiness". Like some sort of theorem... maybe of the form P(H|E) = P(E|H)*P(H)/P(E)

[1] https://en.wikipedia.org/wiki/Problem_of_induction

[2] https://plato.stanford.edu/entries/popper/

That's a good point, I haven't actually spent a lot of time on Popper, though I was amused to find that under some interpretation of his argument, the most extreme varieties of obvious pseudoscience fall under science. In particular I wasn't aware the article even addressed Popper, in particular what's in footnote ii:

>Strictly speaking any method can be made to falsify claims with the addition of a falsification rule. However, the rule must still be shown to be reliable.

At the very least, I think it's shaky to try and apply the same standard regardless of the actual method of the science - since some fields take different approaches to others. That's not to say that all approaches are valid, but that one idea of science in one field shouldn't necessarily need to hold up in other one. Some sciences obviously greatly benefit (or did in the past) from even a simple statement of falsifiability, but to others it seems like a total mismatch for the problem domain.

I hadn't considered Bayes' rule to work that way (my only exposure to statistics is an intro course on information theory as part of an EE degree) but if I'm right, this opens up a discussion as to how risky is risky enough - is astrology still in the picture then?

Without having looked into it, my impression is that the popularity of astrology (and psychics, etc.) is down to the fact that they tend to make broad, vague predictions with a relatively high P(E) that people (emotionally) tend to underestimate. For instance "something bad will happen to you this month" -> "OMG, I stubbed my toe the other day, they were totally right". If we're accurate at assigning P(E), however, then we'll probably end up with a low P(H|E).

The prior P(H) is also arguably trouble. What's a good prior likelihood for the hypothesis that magic sky gods affect your fortune? I don't know, but I'd probably have to start somewhere pretty low.

> I was amused to find that under some interpretation of his argument, the most extreme varieties of obvious pseudoscience fall under science

You can't define science as 'the right answer' because that is unknowable (notwithstanding that there are a lot of things where if we havn't figured out the truth then the universe is set up to radically deceive us).

And if you accept that unknowability as a premise, then pseudoscience is a perfectly valid science. Something can be completely wrong and still be established in a highly scientific way.

Just because something is provably wrong doesn't make it unscientific. It just makes people who believe it provably wrong. They are likely to be stupid, I admit.

Short summary first, long rant follows. Summary: Mathematics and small parts of philosophy and many other disciplines rightly qualify as non-empirical science, but no lesson can or should be drawn from them about empirical science. Empirical questions are fundamentally different from non-empirical questions.

Long rant:

I don't disagree fundemantally but what you say about philosophy irks me as a philosopher. If you looked at a random sample of publications in philosophy you would find out that only a very small percentage of them has the rigour and exactness that we associate with science. I have also met many philosophers, perhaps even the majority of all I've met so far, who would not describe themselves as scientists.

The remaining part of philosophy that adheres to strict standards and uses mathematical method is akin to mathematics and computer science, and perhaps most similar related to formal linguistics and economics in the way it works. I personally consider this part of philosophy a science, it is a kind of applied mathematics, although often speculative and conditional on axioms that are not as evident as in mathematics. Other disciplines have similar non-empirical parts, a typical example is Social Choice theory in sociology, which I would consider a scientific theory, although it is not empirical. In the end, it's applied mathematics.

I agree that mathematics and the relevant rigid and formal parts of other disciplines are non-empirical science, but to these areas the debate about hypothesis testing and the right use of statistics simply doesn't apply. It a fallacy to presume that, because these fields exist, clearly empirical disciplines (or parts thereof) could do without statistics. If a question is empirical, then it has to be addressed with proper quantitative methodology.

This is extremely important to me personally, since I've been in deep disagreement with colleagues for many years about this issue. They work in related disciplines within our philosophy institute and habitually make "qualitative empirical analyses" of texts without treating them as mere precursors to quantitative studies. They see no problems with their methodology, even when I point out to them that the size of their samples would be too small to support the generalizations they make if they did make quantitative studies. To me, this is absolutely crazy, I just can't see how a qualitative analysis of 20 texts could allow you to take these as representative for hundreds of thousands of texts if a quantitative studies of the same texts could not possibly reveal anything useful because the sample size is too small. What's worse, their whole discipline seems to be based on these kind of extremely small scale qualitative empirical studies, plus a very vague mix of fairly imprecise philosophy and common sense. I'm a nice person and get along with my colleagues well, so I won't ever tell them my opinion, but if I'm honest, I'd say that their discipline is pseudo-science or, at best, imprecise, non-scientific philosophy in disguise. (To make this clear, I have no problem with imprecise philosophy and have done it occasionally myself, I just don't think it can qualify as science and not many philosophers consider it as such.)

Long story short, empirical question have to be addressed with quantitative methodology or you get what I'd call "elaborate opinions".

Sounds fair enough.

I do wish that physical scientists, including myself, had more philosophical training though, even if its not science per se, such that we could reliably have original, educated opinions on empiricism, rationalism, etc.

Might I mention my book (Statistical Inference as Severe Testing: How to Get Beyond the Science Wars (Mayo CUP)). https://twitter.com/learnfromerror/status/107197020637171302... I did not know of Hacker news but was trying to trace why my blog errorstatstics.com got its maximal # of hits in over 7 years as a result of my posting their letter on the ASA Statement Now I'll disappear.

I'll have to check it out!

Very interesting, I'll order your book tomorrow!

I feel (as a non-philosopher who only casually reads) as though the question of empirical methods in philosophy comes down in large part to critiques of positivism, operationalism and therefore also the Frankfurt School's aim to synthesise the social, economic, historical, philosophical (if you'll bear with me) and juristic sciences in a total approach rather than restricting to one area. It seems that one could avoid the onus to pursue quantitative methodology by rephrasing the question not as an empirical one. Your example of the twenty texts seems to call for an empirical analysis, but perhaps identifying "common themes in the genre" might not, depending on how the point is elaborated. It's another discussion as to whether such a question should be answered with empirical methods. To me it's a mistake to conflate rigor with empirical analysis, but you seem to agree when you talk about mathematics. Philosophical arguments can be rigorous, but a lot of what counts as some flavor of philosophy (the synthesis mentioned above) is critical and some really does make the claim to be rigorous. So it seems there's now a choice: either to say philosophy is not a science because it is not rigorous (due to some of its elements), or to say that philosophy can be scientific, just as information theory or theoretical physics, or theoretical computer science can be scientific.

Many philosophers don't consider themselves scientists but I think this is at least in part because the mainstream conception of science really is empirical rather than rigorous analysis. In fact, even non-rigorous empirical analysis can pass for science. Depending on the area, I think that qualitative methodology can capture what quantitative can't, or where quantitative would be unreliable or impossible to use without changing the result significantly.

In a nutshell, philosophy can be scientific if a method is pursued rigorously, but a lot of what is called philosophy is critical theory which is important if not scientific (despite using qualitative and quantitative analyses where appropriate). I have a feeling that critical theorists would question the separation we're erecting between science and non-science, especially those who view critical theory as in part an artistic endeavor too.

Analyzing papers in a glam journal for a year isn’t a very good way to judge what counts as science in terms of what is publishable. (“Just because it’s in Nature doesn’t mean it’s wrong!”)

That said, I agree that most of science is about becoming less wrong (choosing the least worst among available models for how observations arose), and that type of model selection problem is what Bayesian approaches excel at.

“Considering the evidence that preceded this study (or trial, or series), what explanatory model does the weight of the evidence best support?”

This need not be a binary decision (witness Bayes model averaging) and it need not be static (Bayesian updating explicitly evolves from a prior, whether flat or subjective). But it better matches the way most people approach research and probability, imho. Extraordinary claims must be supported by extraordinary weights of evidence, and frequentist testing in a vacuum doesn’t enforce this intuition.

Disclaimer: I am a statistician and a part-time Bayesian. Not a zealot, and not a jihadist against subjective inference. I use empirical Bayes procedures when they improve results on an ongoing basis, and I avoid them when the effort is more than the expected benefit can justify. The tipping point has moved over time, as more, faster, and better tools have decreased the cost (effort) to obtain a posterior distribution over complicated models.

I don't think my position disagrees with you. I make a distinction between theories in theoryspace and parameters in parameter space (given a model we all agree on). I define theory as essentially a causal model and explanation of some phenomenon, whereas a model may or may not be causal.

Much of science (particularly the life sciences) is estimating model parameters, and not actually positing new theories. For example, if you're testing a small molecule drug to see if it lowers blood pressure, you want to know how much it affects blood pressure. This is a parameter estimation problem. We all agree that the molecule will get into the body and interact with other molecules, we just don't know how much these molecular interactions will affect blood pressure. Since Nature publishes a lot of life science stuff, it's no surprise that most of their papers are about parameter estimation given an implicitly agreed upon model.

This sort of parameter estimation is important and very different than positing new theories (e.g. quantum mechanics or germ theory). It doesn't make sense to talk about falsifiability of a parameter value. It only makes sense to talk about falsifying an entire causal theory.

"All swans are white." is a theory about swan-ness and should be falsified. "What proportion of swans are white?" is a parameter estimation problem and it doesn't make sense to talk about falsification.

Theories must be falsifiable and subject to falsification. I don't think this precludes gathering corroborating evidence for a theory and then updating your belief about that theory, but it is very easy to find corroborating evidence for theories so falsification is the most practical means of actually finding truth without fooling yourself.

i.e. it is very easy to posit two different theories that make the same predictions in some domain, therefore all evidence in that domain supports both theories. The only practical way to distinguish them is to find another domain where they make different predictions and falsify one or both.

> "It appears all the swans I've seen are white, therefore I posit all swans are white. Oh, wait, there's a black one, nevermind."

That's such a tedious way to to test a theory of swan color. You have to go find swans, which tend to live in hard to reach, dirty, obnoxious environments.

It's much easier to rephrase the hypothesis into a logically exactly equivalent hypothesis, "All non-white things are not swans", and test that instead. I can test that without even leaving my house. Just looking around the room I'm in there are hundreds of non-white things, and none of them are swans.

This situation is called Hempel's paradox, also known as the raven paradox or the paradox of indoor ornithology [1].

[1] https://en.wikipedia.org/wiki/Raven_paradox

The "black swan" story always struck me as ridiculous in confusing a phrase informally indicating rarity with a scientific belief that something doesn't exist or can't happen.

It also confuses mathematical formalism with scientific method; the former is built on rigor, the latter trades exclusively in approximation.

Yeah the article is trying to associate "using p-values" with "The (Only) True Way Of Doing Science™" which to me seems like a fallacy in itself.

Obligatory xckd: https://xkcd.com/2078/