Bayes' rule in Haskell (2007)

Do you really mean that "removing impossibility and renormalizing" is equivalent to Bayes Rule?

For example, imagine that we know a priori that a coin has only two possible biases, both of which are neither 0 nor 1. When I use Bayes Rule to update my belief in the two biases in response to observed data, I don't see how I am removing any impossibilities. It seems much more like I am downweighting the unlikely bias and upweighting the likely bias.

Both, Office haves enterprise as target market and people know this and they set their expectatives in this context; the same thing happens in the Internet where the context is "everyone in the world" and is assumed that most things are popularity-driven (likes, re-tweets, etc).

Because unethical sentences aren't a criminal offence? I mean, if you gave me the first two words from those sentences and asked "what's the most probable next words", I would give you pretty much the same answer. Let me try some more and I'll put my answer next to google's

"lawyers are" -> "greedy sharks" (actual suggestion: scum)

"gays should" -> "be killed" (actual suggestion: be killed)

"marijuana should" -> "be legal" (actual suggestion: be legalized)

"drug users should" -> "get help" (actual suggestion: be shot)

"macs are" -> "better than windows" (actual suggestion: better than pcs)

As you can see, google's opinion on what is the largest cluster of opinions on the subject aligns with what I expect in 4 out of 5 cases. Also note how you're getting both the suggestion that marijuana should be legal and that people who smoke it should be shot. These suggestions aren't google's opinion on the matter, it's what google expects you to think. Thankfully, they're wrong most of the time.

In the end, it's not impossible to write an ethical filter the same way google has a spam filter, the only reason they haven't done it is because there's no pressure to do so. If you don't want to see such suggestions on google, you have two options:

1) Make people not talk about killing gays or how all lawyers are scum

2) Pressure google to add a filter

Also, white people really are partly neanderthal https://en.wikipedia.org/w/index.php?title=Neanderthal&o...

There's a difference between suggesting corrections (like office) and trying to guess your next word though. Google does both, but unless you misspell "blacks are ruining America" it's not going to suggest thataas a correction. Since apparently they expect it to be searched they suggest it as you're typing, but I don't think office does any sort of word prediction as you type? As you said, bing does the same. The standard is higher for office and other actual spell checks because they shouldn't be changing something that isn't meant to say "black people are ruining America" into that. Prediction is entirely different.

+1 it was actually a rather horrifying presentation of Bayes's theorem. I would just explain it in 2 lines:

P(Cause and Effect) = P(Cause) × P(Effect given Cause) = P(Effect) × P(Cause given Effect)

so P(Cause) = P(Effect) × P(Cause given Effect) ÷ P(Effect given Cause)

The end result is an expression executing Bayes' rule and it is used much like an if expression.

As someone who have tried on occasion to get used to Haskell (and failed - Haskell makes me want to claw my eyes out), and who have minimal maths background, I found reading Bayes original paper substantially easier than figuring out the Haskell in that post.

Bayes Theorem can be implemented as an if statement:

  posterior = []
  [observables, unknowns] = simulate_from_model(priors)
  if observables == observed
     posterior.push(unknowns)

posterior now includes random draws from the posterior p(unknowns|observed).

This is my favorite explanation of Bayes statistics since it implements Bayes Theorem without math. It also is the underlying intuition behind the probabilistic programming approach to Bayesian statistics. Rubin (1984) has a great explanation: https://twitter.com/tristanzajonc/status/325120025428119552

The quote wasn't about their technology. It was about their decision making.

Google is famous for data driven decision making.

The quote was from most of a decade ago; probably truer then. (Though Clippy was Bayesian. I read somewhere a claim that its awfulness came from management messing with it.)

Church? I'm not very familiar with the language but I believe it sounds similar to what you're describing. http://projects.csail.mit.edu/church/wiki/Church

Possibly church? http://projects.csail.mit.edu/church/wiki/Church

It's not a Lisp, but there aren't many probabilistic programming languages, so you may be interested in IBAL (http://www.gelberpfeffer.net/Papers/ibal-ch.pdf).

Google lacks customer service. They are not interested in you, but in volume. And you handle volume through statistics.

If you happen to fall on an edge case, they won't really care. Because you fall in the error.

People who try to exploit Google on the margin risk falling the wrong side of that margin. If that happens then Google are not interested in your feeble "but if you look at this way" justifications. This is not abysmal customer service, it is simply not giving abusive idiots the benefit of the doubt. If they don't like it then they can get free service elsewhere.

Try:

An Intuitive Explanation of Eliezer Yudkowsky’s Intuitive Explanation of Bayes’ Theorem

http://commonsenseatheism.com/?p=13156

You probably don't understand it because nobody has taken the effort to explain it to you simply. Everything I have seen on the Internet (especially programming related sources) take a lot for granted and just spread a little code here and there.

Also, a good idea is not "trying to turn your mind inside out", which is what seems to be needed from a lot of explanations I have read.

It is easy, it is simple, it is normal.

I am sorry I do not have a reference handy (because of the above) but those are good suggestions (otherwise I would not be writing them) that have helped me overcome similar "difficulties".

Edit: Cannot help saying it. The 'a priori' and 'a posteriori' and all those terms are just fancy words (which mean what they signal) which scare a lot of people away from something, I repeat, natural.

If you understand procedural languages better, the following article discusses a basic naive bayesian classifier and implements it in PHP :

http://phpir.com/bayesian-opinion-mining

Nate Silver's book The Signal and the Noise explained it to me really simply, and was a great read. Not as immediately helpful perhaps as some of the links in response to your question, but I thought it was worth sharing.

Bayes' Theorem is about updating estimated probabilities after receiving evidence.

For example: I have two bags of sweets. One has 10 sweets, of which 8 are chocolate and 2 are marshmallows. The other has 20 sweets, of which 10 are chocolate and 20 are marshmallows.

I offer to give you a sweet, which I will select at random. Assuming I empty both bags on to the table and choose one at random (uniformly), what is the chance it comes from the first bag? Assume that I do this where you can't see what I choose.

(Answer: 10/30, or 1 in 3 - as these are the ratios of total sweet numbers).

Now, assume that after I've chosen (you still can't see), I tell you that the sweet is a chocolate (This is the evidence). What is the chance now that it came from the first bag?

(Answer: 8/18 - the ratio of first-bag-chocolates out of any-bag-chocolates)

We have now 'updated' our probability estimates based on the evidence ('It's a chocolate'). Important to note here is that the first bag has a higher ratio of chocolates than the second bag, BUT it's still more likely to have come from the second bag due to the second bag having more sweets in total - what we call the Prior probability.

This relationship can be expressed mathematically, which is what Bayes did.

    P(B|A) = P(B & A)/P(A) = P(B)*P(A|B)/(P(B)*P(A|B) + P(!B)*P(A|!B))

(Where P(X) means The probability of X occurring, P(X&Y) means the probability of both X occurring and Y occuring, and P(X|Y) means the probability of X occuring, given that Y has already occurred (or that we assume it occurs))

Let me throw my hat in the ring for this. Tell me how I did.

First, Bayesian formalism shouldn't be hard. Just remember that

    P(A | B) = P(A & B) / P(B)

and it's easy to derive (left as exercise) Bayes's Law. That's quite easy. The hard part, here, is applying it.

Let's say that I have a coin and I run a game where I always bet heads. If I flip tails, you get $1; on heads, I get $1. Let's also say that 1.00% of the people in the world are depraved enough to use unfair coins that always come up heads. You don't know me, so that's a good assumption for how likely I am to be a scumbag. So there's a prior probability of 0.01 that I'm a scumbag running an unfair game, and 0.99 that I'm using a fair coin. Now, we play, and I flip a head. I win. There's evidence (or signal) that is suggestive that it's more likely that I'm a scumbag, but far from conclusive (fair games will turn up heads as well). How much has the probability changed? Well, let's look at the possibilities:

    I'm a scumbag, and flip heads: 1/100 * 1 = 1/100
    -- There's a 1% prior prob., and if I'm a scumbag I'll always flip heads.
    I'm a scumbag, and flip tails: 1/100 * 0 = 0
    -- If I'm a scumbag, I'll never flip tails. 
    I'm using a fair coin, and flip heads: 99/100 * 1/2 = 99/200. 
    I'm using a fair coin, and flip tails: 99/100 * 1/2 = 99/200.

It might help to draw a rectangle for the four possibilities, like so:

      Scumbag    Fair
      (1/100)  (99/100)
    +---------+---------+
    | Scumbag | Fair    |
    | Heads   | Heads   |
    |         |         |
    |  1/100  |  99/200 |
    +---------+---------+
    | Scumbag | Fair    |
    | Tails   | Tails   |
    |         |         |
    |     0   |  99/200 |
    +---------+---------+

Once we flip a head, we can throw out the "flip tails" sections, because we didn't. We flipped heads. That leaves us with a subspace that contains 99/200 + 1/100 = 101/200 of the total space-- we know we're in that space, so we can consider only it-- while the probability mass of the "I'm a scumbag and flip heads" remains 1/100. So the probability that I'm a scumbag posterior to flipping heads is:

    (1/100)/(101/200) = 2/101 ~ 2%

Probability wise, I'm almost twice as likely to flip a coin after a head comes up. That relationship holds very well for small priors, but what if there's a 55% chance that I'm a scumbag? Flipping heads doesn't put that probability to 110%-- first of all, that makes no mathematical sense, and second of all, there's still a chance that I'm fair but just flipped another head. (The actual posterior probability is about 71%.)

For probabilities over 0.5, "twice as likely" doesn't make sense, if you think in terms of probability. What about odds, however? Then, you can derive that "twice as likely" as 55% is 71%.

You can say "twice as likely" if you think in odds, not probability. Odds is probability transformed through p/(1-p), or the ratio between the probability of the event happening and it not happening; it has domain [0, +inf] which means that "twice as likely" doesn't cause that problem. If you think in odds terms, it turns out that odds(I'm a scumbag) exactly doubles each time I flip a head. After the first, that odds number goes from 1/99 to 2/99. If I flip 10 heads, then it's 1024/99; transformed back into a probability it's 1024/1123, or a 91% chance that I'm a scumbag.

In fact, what I think makes Bayesian inference hard is that it involves that subtle context switch between probability and odds.

    If B is observed and there is a known prior _odds_ of A, the posterior _odds_ of A 
    are k times higher, where k is the ratio of the _probabilities_ of 
    (A & B) vs. (~A & B). 

What makes Bayesian inference neat is that you can run it as an online process (you can look at events one at a time). If observed events (signals) are independent, you don't need to process the corpus as a whole, and order doesn't matter. That becomes nice when you factor in concurrency and distributed computation. From these principles, you can also derive logistic regression (multiplication in the log-odds space is a vertical shift along an S-shaped logistic curve) and you start to see why the logistic curve comes up so often in machine learning.

Ok, so what about Bayesian statistics? Essentially, it comes from the idea that:

    (1) There is some prior probability distribution [0] over the possible states S 
    of affairs, which you cannot observe directly but whose relative 
    probabilities you can estimate based on observed events E. 

    (2) For each event E observed, compute unnormalized posteriors according to 
    posterior_unnormalized(S) = prior(S) * likelihood(E | S).

    (3) To compute normalized posteriors, divide each of the unnormalized ones 
    by the sum (or integral) of those likelihoods over all states S. 
    You now have a sum of 1, which makes it a legitimate probability distribution. 

[0] Regarding (1), picking priors is more of an art than a science, which is why some people distrust Bayesian methods. The good news is that if you have a lot of evidence and your priors are reasonable (few assumptions / Ockham's Razor) you will converge to something close to the right answer with enough evidence, regardless of priors.

Now, many actual Bayesian inference methods ignore (3). It's computationally expensive (when there aren't two possible states S, but millions to infinitely many) to normalize and often we don't need to do it. For example, if you're doing classification between two classes Q and R-- with "events" being features of what you're trying to classify-- then you generally only care about which is more likely, not whether there's specifically a 23.75% chance of Q. So the normalization step is generally either skipped (if relative probabilities are all that matters) or procrastinated to the end of the analysis. This "feels wrong" at first but it actually works, and is often better (numerical stability) for the accuracy of the conclusions.

Jeremy Kun's "Math ∩ Programming" blog has two great primers on probability theory:

http://jeremykun.com/2013/01/04/probability-theory-a-primer/

http://jeremykun.com/2013/03/28/conditional-partitioned-prob...

His next post in the series is supposed to be on Bayes' theorem.

The Theory That Would Not Die [1] is a great read. It's a complete history of the fight to legitimize Bayes' theory from Bayes' original experiment through Turing and Nazi subs and beyond. Probably not the quickest way to learn the theory, but worth the read.

[1] http://www.amazon.com/dp/0300188226

For the me the trick was realizing that it's not as complicated as everybody makes it out to be. You probably understood it before you'd ever heard of it.

I while ago I wrote a series of introductory articles that builds from the ground up, using a single coin toss as a running example. Here's the first:

http://blog.moertel.com/posts/2010-12-07-on-the-evidence-of-...

It made some sense to me. The author used the G/M thing to introduce an idea: that it's powerful to think at scale. And then seeks a way to make a thinking-at-scale structure (bayes' rule) as accessible to the programmer as an if statement, by putting it in the language.

That was the only bit of the article that did make sense to me :) I get lost trying to read ML-style languages.

What people are referring to with respect to Google is that if higher-level statistical operations are as easy as an if statement, then this may explain why google takes (presumably amorally) such a statistical approach their customers (because it is easy to).

It's just a quotation that makes you think "what if...". I wouldn't fixate on the company names.

You are thinking about languages that are targeting a single machine at a time. That's the way Microsoft thinks with their history in client-side software.

The implication of the quote is that Google attempts to implement the seemingly intractable algorithms by running them on huge clusters, and returning the results to a large audience that makes the computation worth it. If your language targets a data center then having a language support Bayesian inference might be useful.

I think the original quotation simply used it as a rhetorical device to indicate difference in approach at a more abstract level than literally replacing conditionals with Bayes filters.

As I pointed out elsewhere in this thread, Quantom computer languages literally do this.

Also, if you want to perform Bayseian inference, then it is equally as difficult computationally whether or not it can be expressed as simple 'if' statements. However, in terms of code simplicity, expressing it as simple if statements is easier.