Hacker News Reader

Comment by mikekchar | Hacker News Reader

Interestingly, I find my favourite nitpick: Ockam's razor. The article quotes it as "The simplest solution is usually the correct one". This is a common misinterpretation of it and it's interesting that the quote links to the wikipedia page that has a better statement: "Among competing hypotheses, the one with the fewest assumptions should be selected."

The key problem is equating simplicity with correctness. This is usually disastrous. Once you feel that something is "correct" you stop looking for ways to falsify it. That's the exact opposite for what Occam's razor is used for.

Instead, if you have 2 competing hypotheses (two hypotheses for which the evidence supports both), you use the one with less assumptions. Partly because the one with less assumptions will be easier to work with and lead to models that are easier to understand. But mostly because less assumptions makes it easier to falsify.

Abusing this principle outside of the scientific method leads to all sorts of incredibly bad logic.

View on HN

Very interesting.

From [1]:

Famously, Karl Popper (1959) rejected the idea that theories are ever confirmed by evidence and that we are ever entitled to regard a theory as true, or probably true. Hence, Popper did not think simplicity could be legitimately regarded as an indicator of truth. Rather, he argued that simpler theories are to be valued because they are more falsifiable. Indeed, Popper thought that the simplicity of theories could be measured in terms of their falsifiability, since intuitively simpler theories have greater empirical content, placing more restriction on the ways the world can be, thus leading to a reduced ability to accommodate any future that we might discover. According to Popper, scientific progress consists not in the attainment of true theories, but in the elimination of false ones. Thus, the reason we should prefer more falsifiable theories is because such theories will be more quickly eliminated if they are in fact false. Hence, the practice of first considering the simplest theory consistent with the data provides a faster route to scientific progress. Importantly, for Popper, this meant that we should prefer simpler theories because they have a lower probability of being true, since, for any set of data, it is more likely that some complex theory (in Popper’s sense) will be able to accommodate it than a simpler theory.

Popper’s equation of simplicity with falsifiability suffers from some well-known objections and counter-examples, and these pose significant problems for his justificatory proposal (Section 3c). Another significant problem is that taking degree of falsifiability as a criterion for theory choice seems to lead to absurd consequences, since it encourages us to prefer absurdly specific scientific theories to those that have more general content. For instance, the hypothesis, “all emeralds are green until 11pm today when they will turn blue” should be judged as preferable to “all emeralds are green” because it is easier to falsify. It thus seems deeply implausible to say that selecting and testing such hypotheses first provides the fastest route to scientific progress.

[1] http://www.iep.utm.edu/simplici/#SSH4bi

The second quoted paragraph seems to be attacking a strawman. I don't think it was suggested that we should add silly details to improve falsifiability, but rather remove them. Moreover, it seems like this is a way to choose between existing theories, rather than a way to mutate one theory into a better one.

It's pointing out that the equivalence of "simpler" with "more falsifiable" is not perfect. Nobody is suggesting that we just add silly details for the sake of increasing falsifiability, but suppose two research groups independently arrived at those competing theories. Should we choose the simpler one or the more falsifiable one?

A simpler explanation for valuing simplicity is that a simpler theory requires less storage/processing of the human brain. A more complex theory could explode in complexity so that all of its parts and ramifications wouldn't be easily learnable by a human.

One way to think about Occam's razor is from a probabilistic perspective. Consider the Conjunction Fallacy -- for any two events the probability of both events occurring together is less than or equal to the probability of either one occurring alone. Yet it often makes intuitive sense to people that the more specific conditions are more probable than the general one. (See examples in the wikipedia page: https://en.wikipedia.org/wiki/Conjunction_fallacy)

So the more assumptions you're adding to the hypotheses the more you're getting taxed on the likelihood of it being correct. Therefore it's more likely that the hypothesis with the fewer assumptions to be correct.

Occam's razor, as stated by GP is not about correctness, but tractability. In fact, if you look at it probabilistically, the hypothesis founded on more assumptions is more likely to be correct:

Suppose you have a hypothesis, H, which is based on assumptions A1, A2, ..., Ak. This can be phrased logically as an implication:

    (A1 & A2 & ... & Ak) -> H

Decomposing the implication, we get:

    !A1 | !A2 | ... | !Ak | H

Then

          Pr(!A1 | !A2 | ... | !Ak |  H)
    = 1 - Pr( A1 &  A2 & ... &  Ak & !H)

So, appealing to the conjunction fallacy, assuming that we are adding more assumptions on top, rather than having a greater number of different assumptions, the probability of success actually goes up.

Sorry for the mathematical nitpick here, but that seems to me like a strawman. You silently moved from the original question:

    What is the probability that the hypothesis is correct?

To the very different question:

    What is the probability that the implication "from the assumptions follows the hypothesis" is correct?

Moreover, this different question has a clear answer for every logically consistent theory: It is 1, because it is always true!

Why? Because that's exactly what the theory proves logically. The theory can't tell you whether A1, ..., Ak are all true in the real world, but it does tell you that _if_ these are true, H is also true.

So this is really a typical strawman argument (although maybe unintendedly): It is different from the original question, and it boils down to a trivial but misleading answer.

------------------

Going back to the original question, you'd have to compare the two hypotheses H1 and H2, where the set of assumptions of H1 are a strict subset of the assumptions of H2:

    A1 & A2 & ... & Ak -> H1
    A1 & A2 & ... & Ak & ... & An -> H2

It is clear that:

    P(A1 & A2 & ... & Ak) > P(A1 & A2 & ... & Ak & ... & An)

But from here it is surprisingly hard to conclude "P(H1) > P(H2)", because we have implications and not equivalences. That is, H1 may be true even though the assumptions don't hold. It may be true for different reasons and derived from a different set of assumptions that turn out to be true. Same for H2. So we need to take into account the probabilities for H1 and H2 to be "true for different reasons", which we'll name Pd1 and Pd2:

    Pd1 = P(not(A1 & A2 & ... & Ak) & H1)
    Pd2 = P(not(A1 & A2 & ... & Ak & ... & An) & H2)

To prove the probability variant of Occam's razor, we need to make the following additional meta-assumption: The probabilities that H1 and H2 are "true for different reasons" are very small, and moreover almost identical. So we have:

    Pd1 = Pd2

But with that meta-assumption, we can finally prove the probability variant of Ocamm's razor, as we can now express P(H1) and P(H2):

    P(H1) = P(not(A1 & A2 & ... & Ak) & H1) + P((A1 & A2 & ... & Ak) & H1)
          = Pd1 + P((A1 & A2 & ... & Ak) & H1)
          = Pd1 + P(A1 & A2 & ... & Ak)
          = Pd2 + P(A1 & A2 & ... & Ak)
          > Pd2 + P(A1 & A2 & ... & Ak & ... & An)
          = Pd2 + P((A1 & A2 & ... & Ak & ... & An) & H2)
          = P(not(A1 & A2 & ... & Ak & ... & An) & H2) + P((A1 & A2 & ... & Ak & ... & An) & H2)
          = P(H2)

In short:

    P(H1) > P(H2)

You are right, they are different questions, but the straw man was not intentional, I thought the original phrasing was ambiguous enough that it could be interpreted in both ways ;)

In other words, it was unclear to me what the answer to the question "Are the assumptions part of the hypothesis?" was. If, as I did, we assume that "yes, they are" then I don't think it follows that the probabilities will both be `1`, because we do not have logical proofs for the claims, the implication could only be true in the model (they are not necessarily entailments).

The waters are muddied further still when the hypothesis itself is phrased as an implication.

EDIT

It also strikes me that for your line of reasoning to hold, it is not sufficient that Pd1 = Pd2 are small, but instead `Pd1 = 0 = Pd2`, in order to justify this line:

    > = Pd1 + P((A1 & A2 & ... & Ak) & H1)
    > = Pd1 + P(A1 & A2 & ... & Ak)

Which is tantamount to saying

    (A1 & A2 & ... & Ak) <-> H1
  & (A1 & A2 & ... & Ak & ... & An) <-> H2

Is it not?

EDIT (2)

Ignore that, it is not tantamount, it is a weaker condition.

Maybe the final part of the proof I gave is a bit dense, so here are some additional notes.

First of all, if you know that

    (A1 & A2 & ... & Ak) -> H1

then the following two terms are logically equivalent:

    A1 & A2 & ... & Ak
    (A1 & A2 & ... & Ak) & H1

Also, for the proof which I gave it is sufficient that Pd1 = Pd2. It does not need them to be zero.

Ah yes, I see. I guess I was looking for a place where the fact that `Pd1` and `Pd2` were small, but I guess that's not necessary.

No, the "both are small" was just meant to be a justification for assuming Pd1=Pd2.

> (A1 & A2 & ... & Ak) -> H

This is backwards. It should be

    H => (A1 & A2 & ... & Ak)

It's not "if these assumptions hold, the hypothesis is true". It's "for this hypothesis to be true, these assumptions must hold".

Suppose you have the hypothesis that Bruce Wayne is Superman. Then you see the two of them in the same room together. It's still possible that Bruce Wayne is Superman, but only if he has an identical twin. Your credence that Bruce Wayne is Superman should decrease accordingly.

Umm .. no.

Assumptions are the left-hand side of an implication, by definition. (And the right-hand side is called "conclusion".)

The relevant statement here is not "for this hypothesis to be true, these assumptions must hold".

It is: "for this hypothesis to be derived this way, these assumptions must hold".

There is always the possibility that a hypothesis can be proved in a different way from different assumptions.

Unless, of course, your theory not only proves "(A1 & A2 & ... & Ak) -> H" but "(A1 & A2 & ... & Ak) <-> H". That is, if your theory shows that your hypothesis does not only follow from the assumptions, but is equivalent to its assumptions. That's quite a rare case, though.

If you see Bruce Wayne and Superman in the same room, then "Bruce Wayne is Superman" can only be true if you assume something you didn't have to assume before. It means you should be less confident that Bruce Wayne is Superman.

I'm using the word "assumption" in a natural way. (Also in the way that it's used in Occam's razor.) If you have a definition that says I'm using it wrong, then your definition is silly.

This example is totally unclear to be. Although you declared a clear hypothesis in your very first comment, it is totally unclear what exactly your assumptions are that would lead to this hypothesis.

You can form a hypothesis without basing it on anything. You could for example randomly generate 1 billion sentences and then try to test if they are true.

This is not what is meant by "hypothesis" in Occam's razor, which is about hypotheses that are based on actual assumptions (and using these assumptions to pick a "best" hypothesis).

The initial hypothesis is only a starting point. When building a model where 'mice are smarter than humans' you need to account for all the evidence out there.* Compared to the model where 'humans are smarter than mice' it's vastly more complex or vastly less testable.

* I have heard this defined as hypothetical baggage or implicit baggage. ie. if CO2 is not increasing temperature then why not?

Okay, it sounds like what you call assumptions, I would call "data". Or "background data" or something.

If I think Bruce Wayne is Superman, I might base that on the fact that they're both physically very fit; that one would need to be very rich in order to have the kind of technology that is indistinguishable from alien powers; that Bruce Wayne's parents were murdered, and this could conceivably draw him to a life of fighting crime, which is a thing Superman does.

That sort of thing leads me to form the hypothesis: "Bruce Wayne is Superman".

But that sort of thing isn't what Occam's razor is about. It's about things that we haven't observed to be true, but which would need to be true for the hypothesis to hold. You should prefer a hypothesis that requires fewer such things.

If I see Bruce Wayne and Superman in the same room, then in order for Bruce Wayne to be Superman, he must have an identical twin. I haven't observed him to have one, but that's what the hypothesis requires. Accordingly, my confidence in the hypothesis decreases.

At least in the terminology I'm used to, of mathematical proof, an assumption is a part of the context under which a thing is proven. So having more assumptions weakens the claim (and there is an associated weakening rule [1]).

In other words, the claim "Assuming Q, I prove P" does not mean (to me) that Q must hold in order for P to hold, but rather that one way to show that P is true is to show that Q is true.

[1]: https://en.wikipedia.org/wiki/Structural_rule

Here's how I explain it to laypersons: "given two hypotheses, Ockam's razor tells you which one to test first"

Interesting. Usually, the one I test first is the hypothesis Hn where P(Hn=true)/resources_required_to_test_Hn is the largest.

I meant this as an informal rule, i.e. the "spirit" of the law. I think your suggestion is fairly close, in effect, as you can consider assumptions to imply greater resources (namely in validating them).

AKA lowest hanging fruit first.

Never a bad idea

This is a good point. If you went with the "simplest" explanation then which one do you pick:

1. The patients humors are out of whack. The treatment is bloodletting.

2. The patient has a complex infection involving many physiological systems like immune system, foreign bacteria, gut flora, etc. The treatment is rest and administration of a lab engineered antibiotic for weeks.

Or:

1. The patient is possessed by a Djinn/Demon/spirit and needs an exorcism from a priest/shaman/imam.

2. The patient suffers from mental illness which is difficult to describe let alone treat. Treatment will be years, if not decades, of a mix of therapy, lifestyle changes, and medication.

Sadly, these attitudes still exist, even in the industrialized West. I often visit /r/paranormal because I have a thing for ghost stories and sometimes there's a posting about "possession" which is very clearly about a mentally ill person. When I point this out and ask why this person isn't getting proper care, I'm downvoted to -5 near instantly. Yes, that's right, the guy saying "This isn't a demon, this poor woman needs proper medical help," gets argued with like its the 13th century.

Ockam’s razor can’t tell you which theory is correct, just which one to use when you have more than one theory that accounts the data. It is purely a pragmatic way to rank competing theories.

I find it interesting that now Ockham's razor moved from philosophy to applied statistics. That is, Bayesian statistics can quantify (a more complicated model which fits just as well has, or only slightly better, has lower likelihood) and in machine learning we use it in practice (to avoid overfitting, as too complex models may fit well to the training data, but be suboptimal for generalization).

See also BIC (Bayesian Information Criterium) for selecting models.

My conclusion on reading this discussion is that if a bunch of smart people can't agree on what Occam's razor is useful for, it isn't useful!

>> drzaiusapelord ,scoot

Occams razor does not ask that you accept the simplest explanation. It asks that one take into account as many, and only as many factors as necessary to explain a phenom. It does not promote fallacy or lessen rigour. It is a "loose leash but a tight chain"

As originally defined, it stated: Entities should not be multiplied without necessity(Entia non sunt multiplicanda praeter necessitatem).

Bertrand Russel held the principle in high regard. This quote from Newton encapsulates its application: "We are to admit no more causes of natural things than such as are both true and sufficient to explain their appearances." and simplified for scientists in this form: ""when you have two competing theories that make exactly the same predictions, the simpler one is the better" http://math.ucr.edu/home/baez/physics/General/occam.html

There is a line of scholarship that believes William of Occam (c1287-1347) never made the quote attributed to him. http://www.logicmuseum.com/authors/other/mythofockham.htm

What is termed Occams razor by vog asQuirrel asmad and others is a statistical/logicians derivative not really of concern to most people.

You do make one assumption however, which is: "discussions on the internet, lead somewhere".

Which they don't, therefore you can't say Occam's razor isn't useful. :) funny

I'd say the thing with Occam's razor is that its easier to disprove a simpler answer with less assumptions, thus its easier to place more thrust in it if it does hold up to the same scrutiny as answers that have more assumptions and complexity.

Also, i like Hanlon’s Razor. "Never attribute to malice that which is adequately explained by stupidity." Generalizing here, but people _are_ stupid.

this is a info for test.. sorry

I wrote something on my white board this weekend that is similar in concept: forced efficiencies at random intervals. I hypothesize that systems which have the least moving parts are less likely to suffer "breakage" if the infrastructure on top of which they run is randomly reliable and/or stingy with resources. Also see Gates/Page's law: https://en.wikipedia.org/wiki/Wirth%27s_law

That article is missing the classic formulation: "What Intel Giveth, Microsoft Taketh Away"

http://exo-blog.blogspot.com/2007/09/what-intel-giveth-micro...

Maybe because Moore's law is reaching it theoretical limits?

Actually the law is better translated as: "More things should not be used than are necessary."

Except that you never know what is truly necessary or not since you are making assumptions.

Think it like this:

In the search for a cure for head ache, some guy took aspirin and did a magic rite, and he was cured; another guy just took aspirin and he was cured. Both account can be reproduced, and so far have worked. Therefore when doing analysis you can ignore the bit about magic, while it may be relevant in some mystical sense (the spirits are more happy if you do it, whatever), it is unnecessary to explain the cure from head ache.

I took a class in College where the Professor revolved the class around Occam's razor. Every quiz we would have a word limit associated with each question. It was honestly a quite difficult exercise to concise your answer and remove unnecessary information.

Genuine question: so why do physicists explore a theory that posits 11 dimensions, if the evidence at hand does not require 11 dimensions - i.e. if it is additional assumptions about the Universe beyond what has been collected. (This question refers to string theory.)

Well first you have to come up with at-least two theories that _both_ explain the phenomenon. Then you apply the principle to choose one that's "simpler".

You cant do a 'premature' optimization and not even attempt to piece together a theory that sounds complex, when maybe that is currently the only theory that explains the phenomenon.

I'm not a quantum physicist so take this with a grain of salt, but my favourite analogy is heliocentric vs geocentric orbits: http://i.imgur.com/AReqgfP.gif

The collection of data can have assumptions built in that you are unaware of. If you aren't thinking about the earth moving when measuring the orbits of the other planets the data shows that the orbits are pretty crazy.

Likewise when we measure at the quantum level things seem pretty crazy, things in two places at once, etc. Saying "Once something is small enough it can behave completely differently from what we observe at larger scales" is a pretty big assumption.

String theory trades some assumptions for others to rationalize some of the 'crazy behaviour' at small scales.

This is the major criticism of string theory.

https://www.google.com/search?q=occam%27s%20razor%20string%2...

If you extend OR to theories you should select the one that has the least ad-hoc hypotheses (or the smallest set of auxiliary hypotheses in the Lakatos sense). In other words one could argue that theories get "worse" with more auxiliary hypotheses so from a scientific process point of view if you diligently falsify and force the creation of more auxiliary hypotheses you can weaken a theory enough for a (more elegant) alternative to take its place.

More specifically,

> two hypotheses for which the evidence supports both

If they are supported by the same evidence, then the truth that is being supported must be the same. The simplest hypothesis is therefore the smaller nutshell that captures that truth. The other one is bloated, and bloated information (what this is all about) punishes us with complexity and irrelevance.

Fundamentally, any theory is an abstraction of evidence. Ockham's razor is about the quality of said abstraction.

Still, how do you define "simplest"? This seems quite arbitrary.

It is easy to do in many cases. Take the following two theories:

Rocks falls downwards since masses attracts each other.

Rocks falls downwards due to invisible ghosts making masses attract each other.

Both of these theories have the same amount of evidence supporting them, but one is strictly simpler than the other. If one theory isn't a simple reduction of the other then Occam's razor doesn't apply, but in those cases it is usually possible to make experiments which disproves one of them.

The invisible ghosts are called "gravitons" in mainstream physics community :-)

Less is not arbitrary. Be it fewer assumptions, variables, bits, rules... And these are all abstractions. Abstractions can be counted, so all simple really means is fewer abstractions. Theories tend to refine themselves because ultimately we arrive at one word, and with new evidence, even words will adapt. What was gravity yesterday is graviton today.

In a sentence, "What is?" versus "What do I do?".

Occam's Razor is a useful -- but not fool-proof -- tool for the latter.

It says nothing definitive about the former. At best, it makes a very broad statistical generalization.

If it's actually statistical (measured), and not anecdotal.

Couldn't adding more assumptions also make it easier to falsify? Like if the assumptions obviously lead to inconsistencies or if the added assumption is required for the hypotheses and happens to be easier to test in isolation, then one could throwout the hypothesis while only checking a single assumption.

Fewer assumptions = special cases and exceptions stand out more and are harder to integrate without altering the theory beyond recognition. More assumptions = more moving parts.

Because the idea is that the theory will be tweaked anyway. Nothing's gonna be perfect on the first try.

I think a more accurate interpretation is "the simplest explanation is the likeliest". It follows as a consequence of Solomonoff induction.

"simple" explanations are quite often the ones with the most assumptions so I don't think your interpretation is accurate.

Could you clarify what you mean by that?

Take high school physics, where you assume that there is no resistance, cows are spherical, gravity is exactly the same everywhere, and so on. Assuming all that, you can make a much simpler model than when you don't.

"Simple" here has a particular meaning, related to Kolmogorov complexity.

But it is not about what theory is most likely. If two theories or models generate exactly the same predictions, they are equally true. Occam just says that in this case we should chose the simpler.

But if the theories produce different predictions, then it doesn't matter which one is simplest, the one which most closely match reality is the truest.

In practice we are often in situations where multiple hypothesis explain the data we observe but diverge from each other on data that we have not yet observed and it may be difficult to create the necessary situations to tell between them. In these cases we may still wish to choose a hypothesis to make predictions from.

There are an infinity of possible models to choose from, with most of those models containing no less information than the phenomenon they seek to model. Predictive power is what is important for models; a model that has enough dials to be adjusted to work with any new piece of data might be 'correct' but it is not useful. My favourite example of this is the fact that when the heliocentric model of the solar system was being developed, the geocentric model was providing much more accurate values for the positions of celestial bodies for a long time because it had had years of being tweaked to do so. Initially, it was the simplicity of the heliocentric model rather than its accuracy that was appealing.

If we already know the answer, then we don't need a hypothesis. The only reason to choose is because there are different predictions of the unknown.

Reply on news.ycombinator.com