Bayes's Theorem: What's the Big Deal? | Hacker News Reader

Bayes's Theorem: What's the Big Deal?

blogs.scientificamerican.com

400 pointsNN88259 comments

Interesting that they mention the medical case, when there's some psychological work around the idea that we should present these cases in terms of natural frequencies instead of Bayes' theorem.

The natural frequencies approach is to say "if 10000 people take the test, 100 will have cancer. Of them, 99 will get an accurate positive test, and 1 will have a false negative test. Of the other 9900, 99 will receive a false positive, and 9801 will receive a correct negative. What are the odds that someone who has a positive test has cancer?"

It turns out that doctors and other professionals whose core professional competency doesn't concern probability do terribly when presented with percentages and Bayes theorem, but can handle natural frequencies quite well (here's one quick summary: http://opinionator.blogs.nytimes.com/2010/04/25/chances-are/...).

As is obvious, this isn't an argument that Bayes' theorem is wrong--it's a theorem after all. It's an argument about which types of reasoning people can be easily taught.

It's true that when the question is formed in frequentist terms, the answer is much more intuitive. But is that how the problem occurs in real life? The doctor doesn't see ten thousand people take a test; they see a person take a test, and get either a positive or negative result. The traditional way of forming the problem seems closer to actual experience: 'your patient tested positive. you know how accurate the test is, and how common the disease is; how likely is it that the result is a false positive?'

I'm not quite sure what you're saying. Doctors don't observe probabilities or enormous frequencies. Either way, there are good odds that this is information that someone is communicating to them, not the result of their personal experience.

Doctors observe a result of the test, and know the basic probabilities (in the example, 99% test accuracy, 1% of population have the disease). The problem is that they [often] draw incorrect conclusions from those observations (99% test accuracy and you tested positive? well then you likely - 99% - have the disease, right?).

The question formed as 'your one patient tested positively' is more immediately relevant, I'd think. The correspondence with actual practice is obvious. The question formed as 'out of 10000 ...' could be remembered as a quirk of statistics, but not actually recalled when someone tests positively for cancer.

Of course, doctors do not randomly assign tests to patients. Their prior that a patient has a disease is a lot higher than the background frequency of it occurring.

Getting them to estimate their prior would be interesting.

Note that the frequency with which people who don't have the condition take the test is a major (but not the only) component of a good prior.

It depends. Some tests are applied without prior suspicion, so you deal with exactly the background frequency. With others, the disease in question is so rare that false positives will dominate even if the doctor has serious suspicions. The second case is the reason for the "think horses not zebras" aphorism.

Doing a few explicit Bayesian calculations can help one internalize just how much important is that often forgotten P(A) factor is.

The same applies, by the way, to the antiterrorism security theater - many support it just because they have no intuition (or idea) about base rates.

Except when they do. For example, when 100% of men above a certain age are screened for prostate cancer, or 100% of women above a certain age are screened for breast cancer. Both cases spawned major public health campaigns to encourage screening, followed years later by recommendations AGAINST 100% screening, based on the high degree of false positives and unnecessary treatment.

Other cases that come to mind: -- doctors who offer "full body scans" as a part of an executive physical; you're pretty much guaranteed to turn up something that is 2 sigma away from the population norm, somewhere in the body, on such a scan

-- spinal x-rays for back pain. Doctors almost always find something abnormal, and use that to justify the back pain and treat aggressively. But, we don't really have a good prior; if you x-rayed 1000 people off the street, would we find similar abnormalities frequently?

I certainly agree that thinking in terms of percentages is more familiar. But it's familiarity that doesn't help most people get the right answer.

I think GP's point is that in the case of interacting with an individual patient, the Bayesian conception of probability as a quantification of degree of belief is actually more intuitive than the frequentist conception of probability as a relative frequency of outcomes under repeated hypothetical experimentation.

"Reckoning with risk" by Gerd Gigerenzer is interesting.

https://plus.maths.org/content/reckoning-risk

> The prescription put forward is simple. Essentially, we should all be using natural frequencies to express and think about uncertain events. Conditional probabilities are used in the first of the following statements; natural frequencies in the second (both are quoted from the book):

> The probability that one of these women [asymptomatic, aged 40 to 50, from a particular region, participating in mammography screening] has breast cancer is 0.8 percent. If a woman has breast cancer, the probability is 90 percent that she will have a positive mammogram. If a woman does not have breast cancer, the probability is 7 percent that she will still have a positive mammogram.

> Imagine a woman who has a positive mammogram. What is the probability that she actually has breast cancer?

> Eight out of every 1,000 women have breast cancer. Of these 8 women with breast cancer, 7 will have a positive mammogram. Of the remaining 992 women who don't have breast cancer, some 70 will still have a positive mammogram. Imagine a sample of women who have positive mammograms in screening. How many of these women actually have breast cancer?

If I may rephrase (and steelman) the parent's point:

Reality does not neatly format itself for easy plug-and-chug into your formulas. To appropriately respond to reality, you must be good at recognizing when there's a mapping to a well-tested formula, technique, or phenomenon.

Therefore, if you require problems to be phrased in a way such that that's already done, that means you're not good at that domain; blaming the phrasing of the problem is missing the point.

Indeed, 90% of the mental work lies in recognizing such isomorphisms, not in cranking through the algorithm once it's recognized, and this is a hard skill to teach. (Schools that teach how to attack word problems have to rely on crude word-match techniques to identify e.g. when you want to subtract vs add vs divide.)

> formed in frequentist terms

I suspect that this is simply a sense of scale, with the freq-vs-bayes aspect playing only a minor role.

The normalization that happens when you use percentages and 0.0 -> 1.0 probabilities is often useful, but it can sometimes obscure the magnitude of some relationships. It's easier to understand the sense of scale when you say "1 out of every 100,000 people" (which is easily extended to "a handful of people in a large city"). The same information is presented as "0.001% of the population" requires the reader to do more mental math if they want to understand how many people could be affected.

Knowing how to interpret false-positives and false-negatives is very important, but doctors are busy people who have to memorize a lot of data. It is probably better for patients if they can at least remember if something is "common" vs "rare" when they need to make a quick decision.

Except when

https://en.m.wikipedia.org/wiki/Simpson%27s_paradox

It's a lovely phenomenon, but what are you trying to say?

Maybe that the natural frequency "normatized" for some people may be completely different from the actual frequency that apply to them, because the people that researched the frequency did their grouping badly.

I'm not sure if that's relevant. It seems like either you know the actual frequency for their subgroup, in which case you redo the numbers in the example, or you don't in which case you can't accommodate that regardless of how you calculate.

A natural frequency is different way of expressing and reasoning about the same information as percentages.

Well, me neither. If you get the frequency wrong, you'll also get your priors wrong. Natural frequency does not compose well, but that is another problem.

That is just what I understood from the GP.

It seems likely that this will help some doctors in at least simple cases, but natural frequencies don't 'compose' well if you are doing multiple tests; you can't use them to compute posterior probabilities when combining a half-dozen different tests, at least not without involving impractically huge numbers or fractional people.

Moreover, the trend in modern medicine is towards combining binary/categorical tests with a range of distributions based on age, gender, race, genetic factors, vitals, continuous lab result values, etc. Yes, we can theoretically delegate some or all of that to software, but that is true for medical diagnosis in general; while we're not there, doctors must understand and perform these calculations.

So it seems fine to use natural frequencies to help out, but building Bayesian intuitions early and often seems a better path.

Doctors and professionals are probably not as concerned about Bayes because cancer isn't identically or independently distributed across the population.

Also, endocrinology diagnostic platforms usually don't have identical accuracy rates for positive and negative results, varying by test, machine, controls used, etc.

The reason why doctors and other professionals find it difficult to understand is because they are paid for treating people. As Upton Sinclair observed “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking.

A couple of comments are saying, "no duh" to this article, but those folks likely don't realize quite how many other people are falling into this trap. "Garbage in, garbage out" is only good advice when the person you're saying it to realizes they're putting garbage in.

Do priors just start you off closer to the truth? That is to say, if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?

Does anyone commonly set their priors to be a distribution? Perhaps a range or actually a normal distribution to represent a prior with uncertainty?

Yes and yes and yes!

ETA - caveat to the 2nd question: ... unless you've restricted your prior to exclude the truth.

In my field (Epidemiology), when doing Bayesian analysis, it is very common to set one's priors to be a distribution. Sometimes the point estimate and spread of a previously conducted study or meta-analysis, sometimes merely a uniform distribution with upper and lower bounds ("It is extremely unlikely that the relative risk of disease for this exposure is below 0.01 or above 100...")

It's been argued that frequentist analysis is essentially a Bayesian analysis with a prior distribution centered on zero with bounds from positive to negative infinity.

How does that work? It doesn't sound like a well-defined distribution since the area under the curve needs to be 1.

It's more of a philosophical statement than an actual implementation. Mainly that frequentist analysis begins with the prior "I dunno, could be anything..."

See also: The calculus answer.

It has to do with calculus. If the probability of a given result approaches zero as the result itself approaches positive or negative infinity, then the area under the curve approaches 1.

Imagine 1/2 + 1/4 + 1/8 ... to infinity. The sum approaches 1 as the denominator approaches infinity. With calculus, we can determine that with mathematical methods.

I think he was referring specifically to an unbounded uniform distribution, which is indeed not well-defined.

It turns out that priors don't actually need to be distributions all the time. This often occurs when seeking maximally "uninformative" priors.

Bayesian analysis comes directly from the probability axioms, which are 'frequentist'.

Huh? You mean the Kolmogorov axioms? In what sense are these 'frequentist'?

The third axiom just counts the event space.

That's the right bottom part of the Bayes' rule?

First Qn: yes. But the rabbit is hiding in the 'enough'. Most commonly the poor uses of BT end up with a narratively argued prior + a single suite of evidence.

For a finite set of evidence (particularly chosen by someone with bias), bias + evidence can be arbitrarily far from the truth.

> That is to say, if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?

Yes, with two caveats. First, you can't have assigned zero probability to the right answer. Which means that if you have a probability distribution over hypotheses, and the right answer wasn't one of the hypotheses in the distribution, you're doomed from the start. The Solomonoff prior is a mathematical construct that gets around this problem by including every hypothesis that can be expressed as a Turing machine, but it gets used more as a philosophical token than an actual computing tool because it's unfeasible to use directly. The second issue is that if you have a bad enough prior, "enough additional pieces of evidence" can be an arbitrarily large amount, and there may only be a limited amount of evidence available to collect. In particular, rerunning an experiment over and over can only provide a limited amount of evidence, because of the possibility that some systematic error affects every instance of the experiment.

The prior of Solomonoff Induction is a uniform distribution over the enumerable set of Turing machines?

Ah I dimly remember that it also has an Occams razor built-in, so the prior probability falls off with the lengths of the strings that describe the Turing machines?

This comes to mind: https://en.wikipedia.org/wiki/Aumann's_agreement_theorem

Essentially, two genuine Bayesian rationalists (with some hand wavy preconditions) cannot agree to disagree; ie, they will eventually converge onto the same understanding of an event.

This isn't true though, the problem is there are generally more then 2 possible explanations. For instance let's imagine both of us are using an experimental telescope to observe if some event occurs. We are then looking at four possible scenarios. The telescope could work/not work correctly and the event could happen/not happen. You are confident that the telescope works correctly and also confident that the event will not occur so you give high prior probability to the first and a low to the second. I on the other hand think the telescope is rubbish and the event will almost certainly occur and do the opposite. We sit down and wait and do not observe the event. You then come to the conclusion that the event did not occur and the telescope works correctly, while I come to the reverse conclusion.

One of the "handy wavy preconditions" is that they share priors. This nearly never happens in the real world. Almost all disagreements can be traced to differing priors.

No - the precondition is that agents share common knowledge of each others' priors (not that they have the same priors).

Here's the paper I read: http://www.ma.huji.ac.il/~raumann/pdf/Agreeing%20to%20Disagr...

Here's how it starts: "Theorem: if two people have the same priors..."

Whoops, seems you are correct. I'd misunderstood this, my apologies.

> if you start with any prior, will enough additional pieces of evidence always let you converge on the truth?

That's the general trend: pooling data tends to make priors converge. However, converging priors isn't quite the same thing as everyone converging on the truth.

Statistical expressions themselves are an incomplete explanation, we routinely use assumptions about the direction of causality that aren't captured in them. See Judea Pearl's "Why I am only half Bayesian" paper for a discussion as well as an intro to how his framework approaches such independence assumptions.

> Does anyone commonly set their priors to be a distribution? Perhaps a range or actually a normal distribution to represent a prior with uncertainty?

Almost everyone does this, and the solution is a posterior probability distribution. Most uses of Bayesian techniques that I'm familiar with are based on Monte Carlo simulations, where priors are drawn from the specified prior distribution, processed in some fashion, and result in samples of the posterior distribution.

This is especially helpful when you don't view the problem as having a 'true' single answer with some 'uncertainty' but instead have actual variability in the system, which is described by the posterior distribution.

Thinking about it, I'm really not sure in what instance the prior wouldn't be a distribution. It is, by definition, a probability. I suppose you could have a singular value, where the pdf is a delta function, but what's the point of doing the Bayesian inversion or estimation then? If you have a single value with prior p(x) = 0 or 1, you should end up with the posterior p(x|d)=0 or 1.

So the simplest case is where the prior is a binary: Say p(x=yes) = 0.6, p(x=no)=0.4. Or something like that. It's not a continuous distribution but it's still a distribution. The sum/integral of all cases has to be 1.

Yes, but.

The technical term for the condition you're interested in is consistency. There is a theorem (Doob) regarding the consistency of Bayes estimates. [1]

In simple cases it means that under any non-silly prior your posterior will converge to the truth. E.g. mean of a normal distribution with a normal prior.

The complex cases include things like models that expand when given more data.

[1] http://www4.stat.ncsu.edu/~sghosal/papers/bhurev.pdf

FWIW, it seems to me that a major benefit of the Bayesian approach is to make bad reasoning (in the form of, say, an unreasonable prior) transparent and obvious. I've never heard it claimed that the Bayesian approach was robust to sophisticated idiocy (neither on LessWrong nor mainstream writing on Bayesian methods), except in the narrow techical sense that the posterior asymptotically approximates the likelihood given infinite data (provably true under some assumptions but irrelevant to the objections in the OP).

Yup. Actually, a common theme on LessWrong was realizing that with better reasoning tools you're more able to bullshit yourself, and so you need to be extra-careful.

How do you be extra careful except by developing yet more powerful reasoning tools?

You can't protect yourself in 100% - it would require developing more powerful reasoning tools in an infinite regression. But what you can do is to use introspection, and triple-check your reasoning when it seems to defy common sense or leads you to weird (awful) conclusions.

That's why LW is so big on biases and heuristics by the way - you can treat them as a list of warning signs; if your reasoning seems to match some of them, it's time to take a closer look.

(A thing a lot of people missed since knowing logical fallacies started to become a mainstream thing - you should use the list to check your own reasoning, not your opponent's.)

The problem with trying to rely on heuristics to avoid biases is people often ignore the biases in the heuristics of choice. To continue the example of LW, there are many people there who seem to think highly of IQ test, and who ignore the many issues with them (the Flynn effect an the effect of incentives being a couple examples of the flaws in IQ tests).

Trying to remove biases is great. But there is a problem when someone works to remove some biases, then believes that they are inherently more rational than the public at large, and then uncritically accepts there other biases ("Someone like me who's worked hard to remove there biases must be correct when compared to the biased masses.").

You should probably know what James Flynn thinks of the Flynn effect. He doesn't try to escape the conclusions of an I.Q. test as much as extend them, despite some of the "paradoxes".

Fair enough. It's also worth pointing out that Alfred Binet, generally considered to be one of the fathers (if not THE father) of intelligence testing, felt that the idea of quantitative intelligence testing was severely flawed (and had fairly harsh words for people who believed that there was largely a single measure of intelligence).

Yes, there is that risk, and no doubt many fall for it. Ego / self-esteem issues may be a big part of it. But then again, every worthy goal poses risks. When you fly a plane, there's a greater risk you'll kill yourself than when you stay on the ground, and yet airplanes are being flown and we're reaping great benefits from it.

RE IQ, personally, I'm 100% confused on the topic. I used to believe that Flynn effect is basically people getting better at doing tests, but recently I heard that someone controlled for that and the effect remained. So I don't know. The topic is complicated and most of studies I heard of are the kind of psychology and social science I implicitly assume is mostly bullshit.

The Flynn effect mystery is exactly the same as the mystery that in most countries the younger population is taller than the previous generation. The fact that the Flynn effect only occurs in the bottom half of the intelligence distrubtion is a bit of a clue as to what is the cause.

What's the cause that follows from that it mostly affects the bottom 50%?

What is the cause of the population getting taller?

The brain is just another organ that is affected by nutrition in the same way height is. Improve nutrition and those individuals that are below their genetic potential due to poor nutrition will improve. Guess which half of the population has suffered from poor nutrition in the past?

IQ is held in high esteem because research around g is very good and comprehensive, perhaps the crown jewel of psychology. Additionally most people's knowledge of the Flynn effect is out of date -- recent studies (here's one: http://www.sciencedirect.com/science/article/pii/S0160289615... in fact here's a boatload of references http://www.iapsych.com/iqmr/fe/MasterFlynnEffectreferencelis...) show a rise, leveling off, and then an overall fall from the beginning in performance over the last 40 years rather than the continuous rise (or at least non-decreasing) behavior most people would probably bet on from their layman understanding of the effect. (Additionally ethnic gaps have remained despite controls for everything and it is this unfortunate reality that I think is the reason for so much dismissal of IQ...)

When fairly minor monetary incentives (over $10) can lead to a 20 point increase in scores[1], I'd be wary of reading too much into the tests. As for the Flynn effect, I've read a number of different theories, but there doesn't seem to be any consensus. Given the other issues IQ tests have (like the one just mentioned), I'd be wary about assuming that it doesn't stem from underlying problems with the test itself. Some researchers seem to think it stems from familiarity with test taking in general, which seems to match the general understanding that you will do better on IQ tests if you repeatedly take them, referred to as the "practice effect" (IE, IQ tests at least in part measure familiarity with the test).

[1] http://news.sciencemag.org/2011/04/what-does-iq-really-measu...

This study isn't surprising, I'm unsure what you think it implies. Indeed IQ researchers have been wary themselves and known about motivational effects for decades, along with many other objections to testing like cultural bias and the like that in good modern research are all accounted for -- obviously if I sleep during a test I'll score very low, but if I'm actually awake and care I can increase my score by a huge factor. IQ isn't a perfect correlate, but it's common to reason as if it alone has a predictive power of 0.4-0.6 for various important things, that is stronger than any other single factor we know about. When you add in Conscientiousness (the big five traits being the other contender for psychology's crown jewel, I think), which is about grit, intrinsic motivation, and the like, together with IQ (two factors now, not just one), you get predictive powers of 0.7 to educational success. I would wager that the differences seen in the paper cited by your article are almost entirely accounted for by Conscientiousness, but from the abstract (don't have the full paper) it looks like that was not controlled for at all.

For your pleasure: https://ideas.repec.org/p/nbr/nberwo/15898.html

Spend more time with the data, literature, alternate hypotheses, and so forth.

You don't take anything hammered out by reason alone as sound information. You test reasonable-looking propositions against experience, and until they've stood up against that test by providing accurate predictions with substantial information-content, you take them as provisional.

"The first principle is that you must not fool yourself and you are the easiest person to fool."

- Richard P. Feynman

> I've never heard it claimed that the Bayesian approach was robust to sophisticated idiocy

Alas, nothing is robust to sophisticated idiocy.

"If you make something idiot proof, someone will just make a better idiot."

This made me chuckle.

"The trouble with fool-proof methods is, you're still a fool."

Bayesian seems great when you first see it. It should be obvious how to apply it for something like a card game. The problem is how long would it take you to realize a deck of cards was missing the 4 of diamonds? What if the card was lost 1/2 way though the game? How about on the prior hand?

In the end it's stuck at one level of recursion and all facts are fuzzy.

It's stuck in the reality you define - you can add the last hand, all cards seen so far, and the color of people's jackets to the model if you so please.

Your still stuck with: You observed X/Y. How accurate is your count. How accurate is your estimate of accuracy. How accurate is your estimate of accuracy of your accuracy estimate. ... recursive infinity.

Did you account for the possibility you're dreaming, or in a simulation? Or in a simulation of a simulation and so on? Yes, that's the curse of real numbers. In practice you set a limit of precision required, and a threshold of evidence you need to tell you that the precision should be more precise. You can get quite far with Newtonian physics without considering the corrections of general relativity, you can get quite far in engineering by assuming linearity by only expanding the Taylor transform of a non-linear expression by a bit instead of infinitely, and you can do a lot by assuming probabilistic independence of your card counting from what's going on on Pluto right now.

It is going to take me more than a few minutes to parse this. However, I am relatively certain there is a lot of sarcasm in this snippet and I like it.

> I've been saying this for years, and this is a large reason why I find the LessWrong folks to be almost entirely full of it. Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking.

I assume you're saying LessWrong folks are more prone to miscalculating priors than most. Could you give some examples of this?

The LessWrong folks aren’t obviously better or worse at calculating priors than anyone else. The “problem” is that their hobby is spending their free time considering outlandish scenarios, inventing arbitrary assumptions related to such scenarios, drawing questionable conclusions, and then convincing themselves that because they used logic and math, their analysis must be correct. Plenty of other folks who spend time on similar activities with a less pseudo-rigorous framing end up as conspiracy theorists or occultists; belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans is far from the kookiest thing people convince themselves about.

Oh come on. That's how you're supposed to use math. To aid your thinking. Without it, and considering "outlandish scenarios", we would not have any scientific progress.

Also, this is one strange thing - any time someone asks us (the STEM crowd), "what will I ever use math for in my life?", the default answer seems to be, "it's about having more tools for thinking, and greater clarity of thought; it'll make you smarter". But then, some of us turn around and refuse to acknowledge that people who actually learn math and try to apply it may be getting those promised results. Whether it's LW people, programmers, engineers or scientists, the moment it matters, the default conclusion is that math gives nothing.

What? Who said “math gives nothing”? I spend most of my day building things out of math. I think math and scientific inquiry are basically the most important tools invented/popularized in the past 1000 years.

The lapse here is not math, but rather spending lots of attention on abstract thought disconnected from any kind of reality check. Of course, there’s nothing inherently wrong with speculating sans evidence about the future, it’s generally a harmless hobby. In the best case it makes for fun SF novels. Convincing yourself, still without direct evidence, that your speculation reflects truth implies that something has gone off the rails in the reasoning process, however.

Using statistical analysis to understand real causal relationships in areas we have real data about is damn hard, and even plenty of people who are highly trained as statisticians screw up all the time. Academic fields like comparative politics (to take an example I spent a fair amount of time studying) are rife with poor conclusions drawn from bad analysis. The LessWrong folks are hardly unique in applying logic poorly. But they do tend to tackle more speculative questions and convince themselves more firmly of their conclusions (at least, such is my impression as an outsider).

I think this is a common problem when people working in fields that have somewhat accurate mathematical models look at fields that don't. They often don't realize how hard it is to create an accurate mathematical model for many situations, and assume that the other fields don't have them because the individuals who work in said fields aren't as good at math.

Which is why every so often you'll get things like a physicist spending a couple months studying economics in their free time and deciding that they can now unlock the secret to economics which has eluded economists.

This xkcd comic sums it up well:

https://xkcd.com/793/

The more you extrapolate the more frequently you will need to adjust your future predictions because of errors in your initial measurements. This goes for any kind of extrapolation (for instance: plotting a course on a map), but it goes even more for extrapolating the future from limited evidence present today. Your 'best guess' might be off by many orders of magnitude if the evidence you have today is only loosely related to the future in terms of importance and where evidence may not be nearly as independent in nature as you currently perceive.

This can lead to your best guess based on available evidence being about as good at predicting the future as randomness in spite of all the apparent effort at making the predictions mathematically sound.

But these aren't examples so much as vague caricatures. The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish.

http://rationalwiki.org/wiki/Roko%27s_basilisk

https://en.wikipedia.org/wiki/Pascals_wager

I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes. It's not like LW gives Roko significant air time or serious treatment (EY freaking out and deleting it was partially principle of the thing, partially the fear that somebody might follow this road of thought to come up with something more terrifying and he doesn't want to take the community there, etc) somebody doing this is generally taken as a sign of serious crankery.

(FWIW, I agree with the top parent post that the hyping of Bayes Theorem is one of the LW foibles. At least the presentation of it.)

It's less about the plausibility of the thought experiment, and more about typical online drama and hysteria that ensued, which sort of belies that LW is made up of mortals like you and me. They aren't hyper-rational machines, after all.

In that case, how is it a reply to woodchuck?

"The subject matter that LessWrong considers is certainly unusual, but that alone should not be enough to call it arbitrary, questionable or outlandish."

"But let me tell you about the time LW experienced internet drama."

Doesn't seem particularly germane?

> They aren't hyper-rational machines, after all.

No one claims they are. In fact, if I was to name a single overarching theme of all lesswrong discussions, it would be the fallibility of human reasoning. How is having some reddit-like drama on an open internet forum even relevant?

ps. to belie is to contradict, whereas I think you meant the drama shows that LW is made up of mortals? Just making sure I understood you correctly.

>I don't think it would be fair to malign philosophers because they come up with outlandish scary scenarios that scare people with OCD sometimes.

Actually, that sounds like a fine reason to malign philosophers. In order to consider "outlandish scary scenarios", you must first be quite sure that those scenarios are realistic. If they're not, then you're wasting everyone's time.

And yes, if your expected-utility expressions fail to converge because you believe in taking every Pascal's Wager/Mugging scenario into account, or because you don't believe in time, then you've attempted to take the limit of a nonconverging sequence and no amount of philosophizing will help.

This is still working backwards: a thought experiment that increases your chances of being tortured by a future AI? Surely outlandish! But why? Which premises are truly outlandish, arbitrary, etc? What I see given the premises is no more than what Yudkowsky's already said: ".. a Friendly AI torturing people who didn't help it exist has probability ~0, nor did I ever say otherwise."

(However, I agree that to be genuinely distressed by the thought experiment possibility suggests more is going on psychologically than a rational assessment of unknowns, but this seems to be a minority of the community)

That's the best you got?

It's the most infamous example LW offers.

I've been saying t forever. Thanks for putting it so succinctly. LessWrong is a cult of people who want to be smart and they've essentially found a community in which certain assumptions and hypothetical scenarios combined with mathematical concepts make them think they've found the answer to everything in the Universe.

They're no better than any other cult in my book. The problem is that it's only going to get worse with the advances in AI that are going on. Yudkowski has managed to convince some wealthy people to fund his so called research and we have OpenAI operating in the same waters which somehow gives LW people more legitimacy.

What is the answer to everything in the Universe they think they've found?

I've read a lot of the bigger posts on Lesswrong (http://lesswrong.com/top/?t=all) and none of them are anything like.

No better than any other cult? How are you deciding that? The LW community hasn't killed people. Doesn't cut people off from their family. Does't emotionally/physically abuse people. Etc...

Even if they are a "cult", this puts them miles ahead of other cults, like say, Scientology which has done far, far more harm to people.

I struggle to think in what way LW has harmed anyone at all.

cult |kʌlt| noun 1 a system of religious veneration and devotion directed towards a particular figure or object: the cult of St Olaf. • a relatively small group of people having religious beliefs or practices regarded by others as strange or as imposing excessive control over members.

The veneration of Yudkowski and others in the LW community is more than a bit "religious". So I'd say by definition it's a cult.

LW hasn't done harm to people physically, but what it's done is spawn some very questionable ideas, perpetuate pseudo-science and pseudo-mathematics. The cult leader has no formal training, zero research in peer-reviewed journals and still calls himself a "senior research fellow" in an institute he himself started.

Hell, he even has an introductory religious text - The Sequences and the Methods of Rationality fan fiction (which by the way, he wanted to monetise before the broader fan fiction community stopped him. A clear violation of copyright law).

For more, see: http://rationalwiki.org/wiki/Yudkowsky

I'll quote the section titled "More controversial positions"

Despite being viewed as the smartest two-legged being to ever walk this planet on LessWrong, Yudkowsky (and by consequence much of the LessWrong community) endorses positions as TruthTM that are actually controversial in their respective fields. Below is a partial list: Transhumanism is correct. Cryonics might someday work. The Singularity is near![citation NOT needed]

Bayes' theorem and the scientific method don't always lead to the same conclusions (and therefore Bayes is better than science).[21]

Bayesian probability can be applied indiscriminately.[22]

Non-computable results, such as Kolmogorov complexity, are totally a reasonable basis for the entire epistemology. Solomonoff, baby!

Many Worlds Interpretation (MWI) of quantum physics is correct (a "slam dunk"), despite the lack of consensus among quantum physicists.[23]

Evolutionary psychology is well-established science.

Utilitarianism is a correct theory of morality. In particular, he proposes a framework by which an extremely, extremely huge number of people experiencing a speck of dust in their eyes for a moment could be worse than a man being tortured for 50 years.[24]

Also, while it is not very clear what his actual position is on this, he wrote a short sci-fi story where rape was briefly mentioned as legal.

TL;DR: If it's associated with LessWrong/Yudkowsky, it's probably bullshit.

I cannot judge to which degree these theses are bullshit, but I've found LW a tremendously rich source of thinking tools and I'm convinced that reading or skimming a lot of the sequences have improved my thinking.

Regarding the rape sequence in HPMOR: It's a terribly chosen trope to convey that the fictional society has very different values from ours. Apparently it ties into various parts of the story, so that EY didn't remove it and only toned it down after it was criticized.

> I cannot judge to which degree these theses are bullshit.

Go the link, go to references, read about them. I'll outline the gist: Most of what Yudkowsky says is extremely sci-fi, no real basis in scientific fact, but stretching the current technological progress to the point where his opinions on things (stuff like transhumanism, singularity) can be justified.

What he's preaching isn't science. Certainly not rigorous experimental science. He (along with Bostrom) tends to engage in extreme hypotheticals. Which, sure if you're a philosopher, is fine. But even then, wouldn't you want your work to be judged by like-minded peers? But alas, here has a convenient excuse of being "auto didactic" to fall back on, so he can sit on his armchair and critique traditional education, and his lack of peer reviewed material.

Not to mention, and this is a bit of a pet peeve, I find that most LW people are too self-absorbed, I've literally seen a blog where the person who runs it "warns" the readers that what he writes is too complicated for people to follow. This sort of narcissistic, self congratulatory thinking is what puts me off more than anything. Writing long form posts on the Internet which use complicated words don't make you smart.

http://laurencetennant.com/bonds/cultofbayes.html is another critique. It comes off as bit crass, but stick with it.

> I've found LW a tremendously rich source of thinking tools and I'm convinced that reading or skimming a lot of the sequences have improved my thinking.

There are other ways to improve your thinking. Read books. Read different kind of books, that offer counter point of views. Farnham Street Blog is a good place to start for a list of resources for thinking tools/mental models btw. :)

I will look into Farnham Street's Blog, thanks.

> What he's preaching isn't science.

I don't buy into the necessity that everything has to be peer-reviewed in the old fashioned way. There is peer-review happening in the comments to some extent. I don't take a fancy to dismissing any radical ideas as pseudoscience. It's just the outer fringe of hypotheses that need to be tested against reality, and as long they are approximately humanist, enlightened and don't contradict existing physics without depending on mathematics (or disclaimers), I cannot see anything wrong with it. As a naturalist, I pretty much agree with everything I've read on LW so far, except for the parts I cannot judge (like hypotheses about physics), which I allocate a weaker priors for and a few unconvincing pieces.

> Not to mention, and this is a bit of a pet peeve, I find that most LW people are too self-absorbed, I've literally seen a blog where the person who runs it "warns" the readers that what he writes is too complicated for people to follow.

I have not yet experienced that, but there are also a lot of people on reddit and HN that I don't like, yet I differentiate within these communities between what is valuable and what is not.

> Most of what Yudkowsky says is extremely sci-fi, no real basis in scientific fact, but stretching the current technological progress to the point where his opinions on things (stuff like transhumanism, singularity) can be justified.

At the risk of seeming indoctrinated to you, this is what I believe with high certainty: If Moore's law continues another one or two decades, I think the singularity is a very real possibility. The human brain seems to be nothing more than a learning and prediction machine, nothing what transcends what we can understand in principle. Evolution did come up with complex organisms, but the complexity is limited by biochemical mechanisms and availability of energy. In addition, nature often approximates very simple things in overly complicated ways because evolution is based on incremental changes, not on an ultimate goal that prescribes a design of low complexity. I also think that AI will very likely be superintelligent and that poses a tremendous risk in the 10-40 years to come (on the order of atomic warfare and runaway climate change). By the time someone implements an approximately human-level intelligence, we better have a good idea about how to control such a machine.

> There is peer-review happening in the comments to some extent.

Lol. I guess we don't need college education as well then, there's education happening in the comments to some extent. We don't need traditional means of news, there's news happening on Twitter to some extent. I could go on with analogous line of reasoning.

Don't get me wrong, I'm not 100% in favour of the traditional education model as well, but peer reviews exist for a reason. You and I are not experts in these fields. We rely on the expertise of people who have made it their business and life to study these fields based on a rigorous method. Would you try out homeopathy had it not been rejected completely by doctors and scientists but someone on a forum told you it worked for them? What if someone wrote a very long article with fancy words (like LW tends to do) explaining how and why it works (they exist, I assure you)? Would you try it then?

> I don't take a fancy to dismissing any radical ideas as pseudoscience.

Sure, I'm not saying we should be against radical ideas. That's how scientific progress happens. I'm against LW ideas, for which there is no basis in reality as far as we know based on our current understanding of science.

> I differentiate within these communities between what is valuable and what is not.

Indeed. But I'd rather the community's entire existence not depend on bullshit.

> At the risk of seeming indoctrinated to you, ...

a) Keywords: "If", "Seems" b) Tons of assumptions in that scenario you laid out. If you can't see it, I'm sorry but you're already too far gone. c) Watch some MIT lectures on computer architectures about how the trend of Moore's law has already radically shifted and is flatlining.

Basically, what you've done is precisely the kind of utter crap that LW perpetuates. "If x keeps happening" without providing any reason as to why that would be true. Make some ridiculous simplifications "complexity is limited by ___", nature often does __ because ___. You basically don't provide any rational reason for why you think AI will be super intelligent and even if it were, why that would be risky. You pick numbers out of a hat (10-40 years to come).

Yes, you look pretty well indoctrinated from where I'm sitting. But I hope you see the many (so many) flaws in that last paragraph of yours (it honestly made me laugh out loud :p)

Predicting the future is hard business -- be it the stock market predicting what happens tomorrow, or weather forecast for the next month. It's presumptuous and hella stupid if you think you can predict where Science and Technology will be x years from now.

TL;DR: Stahp.

> Lol. I guess we don't need college education as well then, there's education happening in the comments to some extent. We don't need traditional means of news, there's news happening on Twitter to some extent. I could go on with analogous line of reasoning.

That's a straw man. I did say it's the fringe and it needs to be tested. I didn't say one should replace the other. Peer-review is essentially just mutual corrections, and there are mutual corrections happening in the comments, just not as thoroughly as when it's institutionalized. Most of it is not new anyway, but just summarizes research results and draws logical conclusions from it (for example this [1]). If it wasn't all brought together on LW, I possibly wouldn't have found out about the wealth of knowledge for a long time.

[1] http://papers.nips.cc/paper/2716-bayesian-inference-in-spiki...

> a) Keywords: "If", "Seems" b) Tons of assumptions in that scenario you laid out. If you can't see it, I'm sorry but you're already too far gone. c) Basically, what you've done is precisely the kind of utter crap that LW perpetuates. "If x keeps happening" without providing any reason as to why that would be true.

It's very logical. My certainty referred to the implication, but it is hard, of course, to come up with a prior for that 'if': Exponential progress could continue in various ways, e.g. by invention of more energy efficient chips and by scaling them up, by 3D circuitry, molecular assemblers, memristors, or perhaps quantum computing. There are contradicting studies, so one should put P(Moore's law continues for another 10-20 yrs) at perhaps 50%. So, of course, this is all hedged behind this prior (which I think many people get confused by). The discussion is always concerned with implications which can be made with fairly solid reasoning, by assuming that P(..) above to be 100%.

> Make some ridiculous simplifications "complexity is limited by ___", nature often does __ because ___. You basically don't provide any rational reason for why you think AI will be super intelligent and even if it were, why that would be risky.

That's just a basic assumptions which I find plausible, and which some respectable and knowledgeable persons find plausible too (for example Stephen Wolfram and Mark Tegmark; I am aware that appeal to authority is difficult to argue from, but both have publications which I could also refer to). I agree that mentioning the complexity limitations didn't provide any information because they don't tell us whether it makes it simple enough for us to understand, it merely says that the complexity is not infinite, so I should have left it out entirely. But this is not at all representative for the best contents on LW, it was poor reasoning on my behalf. Bostrom's book Superintelligence gives a pretty good summary about why it is thought to be plausible.

> You pick numbers out of a hat (10-40 years to come).

That's based on estimates of the processing power required for brain simulations by IBM researchers and Ray Kurzweil. Simple extrapolation of Moore's law shows us that we will reach that point roughly between 2019 and 2025. 40 years is just my bet based on what I know about brain models and current obstacles in AI.

I don't understand how you can be so certain that the hypothetical scenarios they imagine can't possibly happen. Even if it really is laughable, we spend a lot of money on laughable research (homeopathy, anyone?), so why is this case so particularly bad?

...Just because we've spent money on one laughable research doesn't mean we should spend it on more, does it? Are you seriously going to argue that?

No, what I'm arguing is that if we only spent money on things everyone agrees are useful, we'd never get anything done.

Not everyone has to agree. That's why we have the scientific method and research institutes. If the current science shows that something is worth exploring in more detail, that some avenues are worth spending money on, spending money on them makes sense.

Bullshit Ideas like AI apocalypse and singularity, transhumanism, downloading brain into a computer do not fit into that criteria.

So how do ideas get to the stage where the 'research institutes' agree they're worth investigating?

It seems to me like you're saying "I don't like these ideas, no one should be working on them". That seems a worse principle then "everyone should work on ideas they find worth investigating".

I also have no idea how you distinguish 'Bullshit Ideas' from non-bullshit ideas without investigating them. Your gut is not that good at distinguishing truth from a blunder.

So I'm a LessWronger and know a bit about the "movement", and think you are misunderstanding what "LessWrongers think". Obviously not all LessWrongers think the same thing at all, but I'm talking about the average position of the people who believe AI safety should be worked towards.

I'd love to explain the basic position, and tell me where you disagree with it. This is the basic position:

1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created.

2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence.

3. "Artificial general intelligence" basically means a machine that is capable of achieving its goals, where the goals and methods it uses to achieve them are general. I.e. not "is able to play chess really well", but rather "is able to e.g. cure cancer".

4. For various reasons, once we have an artificial intelligence, it will likely become much smarter than us. (There are many reasons and debates about this, but let's just assume that since it's a computer, we can run it much faster than a human. If you dispute this point, we can talk about it more).

5. Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world (like we are basically all-powerful from the vantage point of monkeys, and their fate is totally in our hands).

6. (This is, I believe, the main point): Something being "intelligent" in the sense we're talking about doesn't say anything about what its goals are, or about how its mind works. We're used to everything that's intelligent being a human being, therefore the way our mind works is basically the same across every human. With an artificial intelligence, it will work completely differently from our mind. So if we "tell it" something like "cure cancer", it won't have our intuition and background knowledge to understand that we mean "but don't turn half the world into a giant computer in order to cure it".

7. Combine the two points above, and you get the large idea - whatever the goals of the AI will be, it will achieve them. Its goals won't, by default, be ones that are good for humanity, if only because we have no idea how to program our "value system" into a computer.

8. Therefore, we need to start working on making sure that when AI does come, it's safe. Even if we create an AI, the "extra" problem of making it safe is both hard, and we have absolutely no idea how to do it right now. We have no idea how long AI will take, or how long figuring out safety will take, but since this is a humanity-threatening problem, we should devote at least some resources to working on it right now.

That's it, that's the basic idea. I'd love to hear which part you disagree with. I totally understand that not everyone will agree on some of the final details like, e.g., how many resources we should effectively devote right now (you might even claim it's 0 because anything we do now won't be useful).

But I think the overall reasoning is sound, and would love to hear an intelligent disagreement.

> 1. Intelligence can be created, because there is nothing "special"/"magical" about humans, and our intelligence was eventually created.

Human intelligence evolved through a (very long!) series of natural processes, to the best of my knowledge. To say it was "created" implies something closer to a religious or philosophical opinion, rather than something supported by science.

> 2. At some point, humanity will create an "artificial general intelligence". (Since we'll just keep improving science and technology, and there's no fundamental reason why this won't eventually allow us to create an intelligence.

This is hugely debatable. Why is AGI inevitable? Even given great amounts of computing resources, a artificial general intelligence does not just automatically appear, it must somehow be designed and programmed. Fields like computer vision have grown tremendously using techniques like deep learning, but there really isn't any evidence that I know of that a general intelligence is any closer than it was 20 years ago.

Totally agree with your first point, I just didn't want to have too many caveats and nitpicking words. If it's not clear, then of course my arugment in no way implies that human intelligence was "created" by an intelligence - it evolved. Poor wording aside, my statement remains the same.

"This is hugely debatable. Why is AGI inevitable? Even given great amounts of computing resources, a artificial general intelligence does not just automatically appear [...]"

Well no one thinks AGI will appear without anyone working on it, but lots and lots of people are working on it now. And since there are huge incentives to create one, the belief is that more people will work on it as time goes on.

"[...] there really isn't any evidence that I know of that a general intelligence is any closer than it was 20 years ago."

Well, in some sense I agree, in that we still have no idea how far off AGI is. If it's going to happen in 10 years, we should definitely prepare now. If it's 500 years away, maybe it's too early to think about it. But since neither of us knows, wouldn't you say it's worth putting some effort to working towards safety?

In another sense though, I disagree with you that we're not any closer to AGI. As you said jsut the sentence before, fields like comptuer vision have advanced tremendously. While this doesn't necessarily mean AGI is closer, it certainly seems that the fields are related, so advancement in one is a sign that advancement in the other is closer.

Well "is able to e.g. cure cancer" is not actually very general. Which leads to the problem with 2) whats the economics behind creating a general intelligence when a specific intelligence will get you better results in a given industry. Even then specific intelligence is still going to be subject to the good-enough economic plateau that has killed so many future predictions.

Then the problems with 4 on up really concern the speed with which 4 can feasibly happen. The AI goes FOOM doomsaysers seem to think that we'll end up with an AI which is so horribly inefficient that/and it will be able to rewrite it self to be super duper intelligent without leaving its machine (and won't accidentally nerf itself in the attempt) and then that super duper intelligent computer will trick several industries into building an even more powerful body for itself etc... all of this happening before humans pull the plug. no step of which is has anything beyond speculation to support it.

In a general note the full employment theorems mean that even if general AI is economically incentivized there's still going to be dozens/hundred/thousands of different AIs carving out niches for themselves which, given that the earth/universe has limited resources, handily prevents the paper clip maximizer problem. While the future may not need humans it will still be a diverse future.

Yeah, you go off the rails around step 5. "Something being much more intelligent than us means that, in effect, it has almost absolute power over what happens in the world" makes no sense. Since when does intelligence get you power? Are the smartest people you know also in positions of power? Are the most powerful people all highly intelligent?

"whatever the goals of the AI will be, it will achieve them". Dude, if intelligence meant you could achieve your goals, Hacker News would be a much less whiny place.

You're not engaging with the claim as it was meant. In context, no human being has ever been "much more intelligent" than me. Not in the same way that I am "much more intelligent" than the monkey von Neumann.

You might decide that this means edanm goes off the rails at step four, instead. But you should at least understand where you disagree.

I'm still not sure you could assume ultimate power and achieve everything you desired if you were the only hacker news reader on a planet of 8 billion monkeys.

> I'm still not sure you could assume ultimate power and achieve everything you desired if you were the only hacker news reader on a planet of 8 billion monkeys.

I would think it relatively easy for a human armed with today's knowledge and a reasonable yet limited resource infrastructure (for comparison to the situation of an unguarded AI) to quite easily engineer the demise of primate competitors in the neighborhood. Set some strategic fires, burn down jungles would be the first step. "Fire" might be a metaphor for some technology that an AI might master that humans don't quite have the hang of yet that can be used against them. For example, a significant portion of Americans seem way too easily manipulated by religion and fear, an AI-generated televangelist or Donald Trump figure might be a frightening thought.

"Since when does intelligence get you power?" You hit the nail on the head there. Its about I/O. (Just as its about I/O in the original article - garbage in, garbage out). Jaron Lanier makes this point in.

http://edge.org/conversation/jaron_lanier-the-myth-of-ai

"This notion of attacking the problem on the level of some sort of autonomy algorithm, instead of on the actuator level is totally misdirected. This is where it becomes a policy issue. The sad fact is that, as a society, we have to do something to not have little killer drones proliferate. And maybe that problem will never take place anyway. What we don't have to worry about is the AI algorithm running them, because that's speculative. There isn't an AI algorithm that's good enough to do that for the time being. An equivalent problem can come about, whether or not the AI algorithm happens. In a sense, it's a massive misdirection."

As I've said before, the singularity theorists seem to be somewhere between computer scientists, who think in terms of software, and philosophers, who think in terms of mindware, and they seem to have a tendency to completely forget about hardware.

There seems to be this leap from 'superintelligent AI' to 'omnipotent omniscient deity' which is accepted as inevitable by (what for shorthand here is being called the 'lesswrong' worldview) which seems to ignore the fact that there are limited resources, limited amounts of energy, and limitations imposed by the laws of physics and information, that stand between a superintelligent AI and the ability to actuate changes in the world.

1) Define intelligence, knowledge, truth, proof (deductive and inductive)... how do concepts work?, etc. I am not being facetious here. AI is an epistemology problem not a technological one.

2) I agree but we have to solve the problem of induction first but LW/EY are certain that there is no problem of induction. How can one be certain in a Kantian/Popper framework where statements can be proved false but never true?

3a) Here is where we part ways. It is a common assumption that AI implies consciousness but I think that is an unwarranted assumption. Whatever the principles behind intelligence are, we know that consciousness minds have found a way to (implicitly) enact them. It does not follow that consciousness is necessary for intelligence (just the biological manifestation of them) and I think good arguments exist to think that they are not correlatives. If they are correlatives then it will be easier to genetically design better babies, now that evolution is in conscious control, than to start from scratch.

3b) Goals, values, aims, etc. are teleological concepts that apply to living things only because they face the alternative of life or death. Turning off your computer does not kill it in the same sense that a living thing that stops functioning dies forever. 3a) & 3b) diffuses all the scary AI scenarios about AI taking over the world. It does raise the issue of AI in the hands of bad people with evil goals and values, like the dictator of North Korea who now apparently has the H-Bomb. This is the real danger today.

4) I agree. Computer aided intelligence will allow us to accelerate the accumulation of knowledge (and its application to human life) in unimaginable ways. But it will be no more conscious than your (deductive) calculator.

5) Non Sequiturs. Possibly psychological projection of helplessness or hopelessness.

6) As the joke goes, we can always unplug it.

7) Granting your premises then the goal of LW/EY should not be AI but the scientific, rational proof and definition of ethics but their fundamental philosophic premises won't allow it.

8) For me the threat is bad, evil people in possession of powerful technologies.

>Granting your premises then the goal of LW/EY should not be AI but the scientific, rational proof and definition of ethics but their fundamental philosophic premises won't allow it.

That is the goal of MIRI, the organization that EY founded, and is a frequent topic of discussion on LW

MIRI’s three research objectives are, at present:

•highly reliable agent design: how can we design AI systems that reliably pursue the goals they are given?

•value learning: how can we design learning systems to learn goals that are aligned with human values?

Not exactly what I meant. What are these human values (for humans not robots) and how do you prove they are rational and scientific? Their goal is to design AI that will accept human goals/values without defining a rational basis for those human values.

(5) isn't convincing: pull the plug of the computer hosting the AI: it's "dead".

> because they used logic and math, their analysis must be correct

> belief in AI overlords ruling humanity, the technological singularity, cryogenics, or impending 1000-year human lifespans

I don't think anyone on LW believes these are 'facts' that are 'correct'. LW commonly thinks of these ideas as risks / opportunites that might happen (except for the 1000-year human lifespans, which is a new idea to me), and that it's probably worth investing minor amounts of money in case it does. In case of cryonics, that's about $20/month for insurance that covers it; in case of AI safety, that's a couple people doing research on the problem, and some amount of money sent their way.

The way you phrase it seems like LW people are certain that cryonics will definitely let them be revived after death. That's definitely not the case - in fact, IIRC, on a survey a year or two ago LWers subscribed to cryonics assigned lower probability of it working than ones not subscribed. It's not a cargo-cult.

Substitute “plausible” for “correct” if you want to give them the benefit of the doubt. Either way it all so speculative as to be basically pure fiction. It reads very similar to me to various “scientific” defenses of particular religious traditions.

Again, as I said, I don’t think there’s anything inherently wrong with this. Little communities of people should do whatever harmless hobby is fun for them.

I just don’t find it very interesting or insightful.

If people want to spend time and attention and resources on existential risks, how about the ones which are clear and imminent, like wealth inequality, the retreat of world democracy and increasing power of entirely unaccountable and amoral multinational corporations, or global climate change, e.g. http://www.esquire.com/news-politics/a36228/ballad-of-the-sa...

> Their inability to come up with accurate priors is completely lost on many of the folks who follow this kind of thinking.

Mis-specified/faulty priors are absolutely a possible pitfall of Bayes, but Jeffreys Priors are a means to circumvent this: https://en.wikipedia.org/wiki/Jeffreys_prior

Another thing not mentioned in the article is that Bayes DOES converge to the frequentist result in the limit of big data.

Except that at least with Bayesian methods the prior is explicitly laid out.

Frequencist methods when they are used to make predictions and get useful information out of experiments have hidden implicit priors that bias inferences in opaque ways.

The very honest frequencists will admit that their procedures are only rejecting hypotheses so small as to have no practical utility. Others will use weird and dishonest doublespeak where they call rejecting an insignificantly small hypothesis "statistical significance".

But I suppose they do redeem themselves a bit with the wording "null hypothesis" which candidly conveys the sense of having significantly rejected _nothing_.

If frequentism is so terrible, why has the field of statistics had so much success over the last century or so.

Just because something is better than nothing, doesn't mean it's optimal.

Bayesianism is a 'grand unified theory of reasoning' that all of science be should be based on assigning (and updating) probabilities for a list of possible outcomes; the probabilities are supposed to indicate your subjective degree of confidence that a given outcome will occur.

Contrast this with an alternative conception of rationality as espoused by David Deutsch.

David Deutsch in his superb books, 'The Fabric Of Theory' and 'The Beginning Of Infinity', argued for a different theory of reasoning than Bayesianism. Deutsch (correctly in my view) pointed out that real science is not based on probabilistic predictions, but on explanations. So real science is better thought of as the growth or integration of knowledge, rather than probability calculations.

So what's wrong with Bayesianism?

Probability theory was designed for reasoning about external observations - sensory data. (for example, "a coin has a 50% chance of coming up heads"). In terms of predicting things in the external world, it works very well.

Where it breaks down is when you try to apply it to reasoning about your own internal thought processes. It was never intended to do this. As statistician Andrew Gelman correctly points out, it is simply invalid to try to assign probabilities to mathematical statements or theories, for instance.

Can an alternative mathematical framework be developed, one more in keeping with the ideas of David Deutsch and the coherence theory of knowledge?

I believe the answer is yes, and I am going to sketch the basic ideas for such a framework.

The basic idea is to separate out levels of abstraction when reasoning (or equivalently, levels of recursion). In my proposed framework, there are 3 levels, and each level gets its own measure of 'truth-value'. All reasoning must terminate in a Boolean truth value (True/False) at the base level but the idea is that different forms of reasoning correspond to different levels of abstraction.

1st level: Boolean logic (True/False)

2nd level: Probability value (0-1)

3rd level: Conceptual coherence (categorization measure)

For full reflection, you need three different numbers: a Boolean value (T/F) at the base, a probability value (0-1) at the next level of abstraction, and an entirely new measure called conceptual coherence at the highest of abstraction.

As a rough working definition of conceptual coherence, I would define it thusly;

"The degree to which a concept coheres with (integrates with) the overall world-model."

It should now be clear what's wrong with Bayesianism! It only gets us to the 2nd level abstraction! There is not just uncertainty about our own knowledge of the world (probability), there is another meta-level of uncertainly; uncertainty about our own reasoning processes, or logical uncertainty. Bayesianism can't help us here. Conceptual coherence can. Lets see how:

All statements of the form:

‘outcome x has probability y’

can be converted into statements about conceptual coherence, simply by redefining ‘x’ as a concept in a world-model. Then the correct form of logical expression is:

‘concept x has coherence value y’.

The idea is that probability values are just special cases of coherence (the notion of coherence is more general than the notion of probabilities).

To conclude, conceptual coherence is the degree with which a concept is integrated with the rest of your world-model, and I think it accurately captures in mathematical terms the ideas that Deutsch was trying express, and is a more powerful method of reasoning than Bayesianism.

Good article.

I'm only a bit disappointed that the author seems not to realize that Bayes' theorem is just a simple consequence of probability theory, and should be attractive not because "maybe the brain is Bayesian", but because it is based on sound set-theoretic and analytic principles. If Bayes' theorem is false, so is probability theory, and so is nearly everything we know about probability.

Edit: Here is a good explanation of the theorem that makes it visually clear how only set theory is involved in deriving it: https://oscarbonilla.com/2009/05/visualizing-bayes-theorem/

I guess a hard-line frequentist (if such a person exists) would counter that you can't assign probabilities to hypotheses or fixed parameters. Then Bayes's theorem (and every other statement about probability) is true only when applied to statements about how often a certain event will occur.

But of course, most people do assign probabilities to hypotheses and fixed parameters, even if only informally. Bayesian probability theory is an attempt to formalize that kind of intuitive reasoning.

I think it's strange this sudden comeback of a theory that was dismissed more than 70 years ago by Fisher and many others, but no one, as far as I know, cares to explain why Fisher was wrong and why the theory is right. It makes me very suspicious, to be honest.

Actually, many Bayesian textbooks cover that. Try Jaynes - Probability Theory: The Logic of Science.

>I guess a hard-line frequentist (if such a person exists) would counter that you can't assign probabilities to hypotheses or fixed parameters. Then Bayes's theorem (and every other statement about probability) is true only when applied to statements about how often a certain event will occur.

If you model "thinking" and "believing" as sampling in probabilistic programs (which they do in some schools of cognitive science), then Bayes' Theorem becomes a theorem about how often certain execution traces occur when the sampling program is run with fresh randomness. You then need none of the weird metaphysics associated with "subjective Bayesianism".

I have heard of people genuinely saying such a thing. Fortunately, it is nothing but an empty redefinition of the word "probability". In fact, rational degrees of belief in hypotheses do follow the kolmogorov axioms (as shown by eg. Cox's theorem or the VNM theorem), and bayes theorem does therefore apply. Whether or not someone refuses to call that "probability" makes no difference.

All probabilistic tools are accurate in theory, otherwise we wouldn't use them. Bayes' theorem is no different from e.g. the t-test in that regard. The question of whether it's worth using Bayes explicitly rather than other tools is and should be a question of whether we find it aligns with our understanding and helps us think more clearly.

Just because a theorem is true doesn't mean you can't misuse it or that you don't need to do some work to map it to reality.

For example, the Banach-Tarski theorem is solid, but that doesn't mean you can start a business making golf balls by buying one and then endlessly replicating it.

Certainly. Just because you can name a theorem doesn't mean you can derive it either. The article had no actual computation or derivation of the theorem. Instead, it talked about beliefs and other things that don't really exist (in regards to the computation of a probability value).

Given that

  p(A and B) = p(B and A)
  p(A and B ) = p(A) * p(B|A)

we have:

  p(A) * p(B|A) = p(B) * p(A|B)
  p(B|A) = p(B) * p(A|B) / p(A)

So can frequentism.

Many investigators in parapsychology who were sincere and intelligent appear to have based their career on the incorrect use of frequentist statistics.

And it's not just them. Ernerst Rutherford, who discovered the atomic nucleus, "If your experiment needs statistics, you ought to do a better experiment." In the 1990s I was a physics grad student and I think none of the professors had ever heard of the idea of a parameter estimator so we had a bunch of ad-hoc ways to fit power law coefficients that gave different answers and no way to judge goodness of fit that was thought out at all.

One postdoc in my lab suffered through a difficult job market before finally, after a decade of anxiety and uncertainty, got a tenure track position and eventually wrote a paper on how to fit power law curves... in a statistics journal.

And this was in a good department with people in which I was proud of both the teaching and research going on.

Rutherford should just be happy he discovered the last reasonably tangible piece of matter humanity will see for a long time, if ever.

I'm a little confused. Are you saying that a tenure track prof wrote a paper on how to evauluate fitted power law curves? Was it something else besides least squares? Because I can't possibly see this getting accepted to a statistics journal.

It is far beyond least squares as it was and is practiced

There already was stuff in the stats literature in the 1990s that was much better but people in the physics community (such as myself) were not aware of that literature. On the other hand, stats people were not particular aware of the way power laws were occuring in physics.

I saw things that did not add up ten years earlier and Mark Newman did too but we were both so caught up in the rat race, consensus reality, collective delusion, whatever you call it that I left physics before I could address the problem and he suffered through years of bullshit before he could find the time to do something about it.

Watching Mark write great papers, write great book chapters and suffer from tremendous anxiety over his career was a big reason why I left.

Fitting distributions is a little bit different than the usual model fitting scenario where least squares is appropriate. Sometimes people do things like construct a histogram and then do a least squares fit to the bin heights, but that procedure doesn't satisfy the usual assumptions that justify least squares (observations with independent, equal variance, Gaussian errors).

Cosma Shalizi has written some interesting posts on this subject, and also published papers in statistics journals:

http://bactra.org/weblog/491.html

Remember what a parameter estimator is.

If you sample 100 values out of a very large pool and add them up then divide by 100 what you get is not the mean of the distribution, but an estimator of the mean of the distribution as you would get a slightly different answer if you picked a different 100.

Often estimators are simple formulas like that (they are for power laws) but there are subtle details, for instance to get the standard deviation in that case you might think you divide by N (100) but you really should divide by N-1 (99).

Back in the 1990s myself and the people around knew some popular statistic formulas but not the concept of estimating a parameter.

And of course its not physics. Social scientists and life science people tend to take a course on statistics but it is fair to say that the median paper in those fields has some mistake in how they do statistics.

> If you get tested again, you can reduce your uncertainty

I've always been bothered by statements like this about medical tests. This assumes that false positives are statistically independent. But isn't it more likely in general that false positives would be highly correlated in individuals, test administrators, or labs? E.g. If the same person takes the same test from the same doctor and sends it to the same lab, it seems extremely unlikely that the results will be independent. And to me at least, it seems highly likely that a false positive will correlate with some aspect of the individual's biological (e.g. some similar substance in the blood to what's being tested), and as such even using a different doctor/lab would not be all that likely to ameliorate this issue.

These are questions that themselves would need to be answered with scientific research. In the absence of real empirical data, I'd be uncomfortable with saying it is "more likely in general that false positives would be highly correlated." Without data, we don't know how likely it is.

Which is why the author of the article needs to mention that the second test is independent of the first.

I think this is a good question - I hope someone who actually knows about this stuff chimes in with some answers.

In the mean time as a thought experiment, going with a blood test example:

* levels of the compound being tested may rise or fall naturally, and the test checks for a certain concentration or above. * the patient may have the condition, but not have had it long enough for markers to have risen to "trigger" levels. * The patient may not exactly follow pre-draw instructions on eating etc, skewing results * in the case of false negative: the person's immune response may be temporarily suppressing the marker * in the case of something like cell counts - this sample could just be a random local variation * The tech or doctor or whoever could randomly make a mistake on one sample, but not on each sample.

And so on. My devil's advocate point here is that the patient, doctor, lab, and so on are not deterministic code - there are a lot of random inputs in the entire process chain.

That's a nitpick on a correct statement. Unless two tests are always perfectly correlated, you will reduce your uncertainty. They don't need to be independent.

Exactly, unless the false positive is causal some how, testing again will reduce the uncertainty some amount. Though that amount may be less than if the false positive isn't completely independent.

Sorry, I should have provided a more complete quote:

> If you get tested again, you can reduce your uncertainty enormously, because your probability of having cancer, P(B), is now 50 percent rather than one percent. If your second test also comes up positive, Bayes’ theorem tells you that your probability of having cancer is now 99 percent, or .99.

This statement is incorrect if the results are correlated at all.

You could also look at it the other way: Using the same doctor and lab and procedure is the best way to eliminate a false positive, because if the cause was external, then the cause may not be repeated. But if you went to a new lab/doctor/whatever, you've now introduced new variables that could cause a false positive on top of whatever already caused it.

Given that it could go either way, it makes sense to think of each one as independent.

Now if you really wanted to take advantage of Bayes, if you got a positive test then you should get two more tests, one with the same lab and one with a totally independent lab (or even if you got a negative test, assuming your first Bayes run gives a 50% confidence)

Down mods for this?

More information allows for better reasoning. Repeating the same test and also doing a "more independent" test are the minimum next two things you should do given a positive result on a single test.

I think the idea is that, if you retake the test, your system will be in a different state (our organism's biochemistry fluctuates naturally). So if something in your system caused a false positive, there's a chance it won't be there when you retake the test and you will get a true negative.

Basically, you're thinking of your blood composition as stateless, when it may also be counted as an "external factor".

Ya that is definitely the idea. The question is is that idea correct. It seems almost certain to be the case that sometimes it's correct and sometimes it's not, but i'm not aware of any research into which cases are which, and I think that is a huge problem.

Referring to an appearance of Bayes' theorem on Sheldon's whiteboard:

> Bayes’ theorem has become so popular that it even made a guest appearance on the hit CBS show Big Bang Theory

That's not a sign that it has become popular, any more than the appearance of the standard human pedigree notation used by genetic counselors on the same whiteboard indicates that standard human pedigree notation has become popular among the general public. It's simply a sign that BBT takes care to make the whiteboards generally scientifically sensible.

They have a UCLA physicist [1] consultant who works with the producers, writers, prop people, set decorators and others to try to make things scientifically accurate.

In this particular episode, both Bayes' theorem and the genetic information are there because Sheldon was trying to figure out his chances of surviving until technology reaches the point that he can transfer his mind to an AI and what he would have to do to improve those chances. So both those things were on the whiteboard because they are things that would be perfectly reasonable to find on the whiteboard of someone doing that.

They consulting physicist had a blog [2] where he covered most episodes and talked about the whiteboards and other scientific content. Here is the entry for the aforementioned episode [3].

[1] http://bigbangtheory.wikia.com/wiki/David_Saltzberg

[2] https://thebigblogtheory.wordpress.com

[3] https://thebigblogtheory.wordpress.com/2010/09/30/s04e02-the...

P(popular | appeared on BBT) < 1

> In many cases, estimating the prior is just guesswork, allowing subjective factors to creep into your calculations. You might be guessing the probability of something that--unlike cancer—does not even exist, such as strings, multiverses, inflation or God. You might then cite dubious evidence to support your dubious belief. In this way, Bayes’ theorem can promote pseudoscience and superstition as well as reason.

Oh please. You can do plenty of psuedoscience and superstition with good old frequentist statistics. And of all the people you could pick to represent Bayesian statistics, instead of I don't know, Andrew Gelman or someone, the author picks... Eliezer Yudkowsky? If nothing else, this provides inspiration for me to quit procrastinating on my "ASK ME ABOUT ROKO'S BASILISK" novelty t-shirt idea.

> Oh please. You can do plenty of psuedoscience and superstition with good old frequentist statistics.

That's not really the point. The article is simply saying that Bayesian methods are not a silver bullet; it's not saying that other methods of statistics are free from problems.

> The article is simply saying that Bayesian methods are not a silver bullet

But it's not really saying it in passing; the headline is taken from that paragraph. At that point it almost feels like a takedown of a strawman Bayesian.

I suspect Eliezer is targeted specifically due to his tongue-in-cheek presentation of understanding Bayesian statistics as being initiation into a cult. Also due to the author's familiarity with the topic likely specifically as a result of Eliezer's efforts to popularize the subject and his association to him resultantly.

Sure, but who cares? There's an entire field of statistics called "Bayesian statistics", who do actual math and statistics and don't give a damn about novelty T-shirts.

Roko's Basilisk has decided to punish you by having allowed others to create the shirt ahead of you.

http://i.imgur.com/XlydDcK.jpg

https://teespring.com/ask-me-about-roko-s-basilisk

The second one is actually mine, I didn't promote it because I was unhappy with the font.

Yudkowsky has been one of the most instrumental and highly regarded figures in promoting Bayesian epistemology, among other woo that engineers and programmers are oddly susceptible to. Nothing wrong with singling him out.

I agree with pretty much everything you say here, but then I think the article conflates Yudkowsky's Bayes-flavored woo with what most people mean when they talk about Bayesian statistics.

> If nothing else, this provides inspiration for me to quit procrastinating on my "ASK ME ABOUT ROKO'S BASILISK" novelty t-shirt idea.

Can you tell us the story about that party when you got so drunk and started yelling in anger at a poor random dude? Except you weren't drunk at all and the dude behaved like an asshole, but nobody cares about the truth since you're a nerd and you look cute when you're sad that we're telling stories about your drinking problems.

That's basically what happened there. I wish people stopped with this Roko's Basilisk nonsense.

Can you unpack that strange analogy a little, please?

There was a thread on LW about potential information hazards[0] - in particular, a kind created by the special flavor of decision theory which was being developed there. This was one of the "peculiar out there" kind of stuff LW people like to play with. Here comes Roko, who says that he created a potential hazardous piece of information - a thing that if you know it, you're fucked - that could lead to people suffering, and then posts it right there. Eliezer deleted the comment, claiming Roko is being irresponsible (and wrong, but still irresponsible if he believed that what he posted was a hazard), and the topic of what became known as Roko's basilisk was banned from discussion for some time.

Eliezer's argument was that Roko behaved unethically by posting something he claimed to believe will hurt people who will read it. That's irrelevant to whether the post itself was an information hazard or not. Eliezer actually later claimed it wasn't, that Roko was wrong. But the whole situation ended up being used to prove the point that LW people are loonies who invent and believe crazy things.

Hence my analogy, in which one person is rightfully angry at another, but ends up being painted as having issues instead.

[0] - https://wiki.lesswrong.com/wiki/Information_hazard

> my "ASK ME ABOUT ROKO'S BASILISK" novelty t-shirt idea.

Why should we take you seriously when you resort to such cheap attacks?

Read through most of the article just for "people can abuse priors"? Come on. Anything, used wrongly, can promote superstition and pseudoscience.

I think you're missing the broader argument, which is using 'mathy' concepts to dress up poor reasoning. Obviously priors matter, but what matters most of all is how good/complete your evidence is. Using a mathematical formula to lend credence to weak evidence (through liberal use of assumptions) is a hallmark of pseudoscience. The same could be said of many of the abuses of statistics and Bayes theorem is merely one good example of this.

Is using mathy concepts to dress up poor reasoning worse than not using anything to back up your reasoning? At least you can point out exactly what's wrong with the mathy reasoning.

A colleague of mine says 'Sometimes pulling numbers out of your arse and using them to make a decision is better than pulling a decision out of your arse'

"Is using mathy concepts to dress up poor reasoning worse than not using anything to back up your reasoning?"

I believe so. If your belief is baseless, or based on flimsy evidence or simple bias, it's best if that's obvious. Dressing up weak reasoning to seem stronger is a form of lying. It's what we call sophistry. A big part of the problem is that for a lot of people don't understand the math well enough to point out what's wrong with it or have a bias towards explanations that seem complex or sophisticated but really aren't.

It's true that sometimes we have to make a decision based on poor or no evidence but it should be clear that that is the case when that is the case. Dressing up the argument only obfuscates that.

Honesty is an ultimate issue here. If my reasoning is shoddy, but I plug it into some math apparatus, then it'll likely make my problems obviously wrong. If my reasoning is very inaccurate and the data uncertain, being precise about it can at least make the results salvageable. Scott Alexander argues for this position quite well in [0].

Humans can lie with statistics well. But they can lie with plain language even better.

[0] - http://slatestarcodex.com/2013/05/02/if-its-worth-doing-its-...

"If my reasoning is shoddy, but I plug it into some math apparatus, then it'll likely make my problems obviously wrong."

That's pretty clearly untrue. I remember reading a study recently where the p value was less than .01 or something like that but where the experimental design was clearly flawed. The correlation wasn't the correlation they thought they had. But because the math looked good and it was easier than actually reviewing the experiment, it was tempting to take the study on face value.

I've read Scott's essay before and I understand his argument, but I don't think it works. While, you might be able to avoid some bad reasoning simply by being more systematic, you can also strengthen bad arguments with a faulty application of statistics. What Scott doesn't do is provide an analysis of how often each of these things happens. I'd argue that for each time a quick application of statistics save someone from a bad intuitive judgment, a misapplication of statistics is used to encourage a bad judgment at least one time if not more.

Understand that my argument here is not that one should never use statistics or even Bayes theorem, but that a naive or lazy application can be worse than no application.

I see your point and I agree.

For myself, I try to limit myself to the mathematical apparatus I feel comfortable with. I know that if I were to open a statistics textbook, I could find something to plug in my estimates and reach a conclusion, and I'm pretty sure the conclusion would be bullshit. I learned it the hard way in high school - I remember the poor results of trying to solve math and physics homework assignments on topics I didn't understand yet. The mistakes were often subtle, but devastating.

> 'Sometimes pulling numbers out of your arse and using them to make a decision is better than pulling a decision out of your arse'

Agreed! Leaving the pseudoscience example aside - since there are strong emotions involved - we can clearly see that it is indeed useful and necessary to take decisions under uncertain/incomplete information. This is advantageous whenever the cost of inaction is expected to exceed the cost of backtracking a less than perfect decision, which often is the case.

Let's say... project management. IF you take the time to find out that your project requires 100 tasks, 30 of which lay in your critical path; you can argue if each task will take one day or one week to complete, and you can debate whether adding a 3rd or 4th member to the team will significantly speed up the completion date or not. But you will definitevely be in better shape than if your PM just cook up some 5-page-spec overnight and commited to have it running in beta test by the end of the month before even anouncing it to the team...

Which itself will be better than having all your potential contracts snatched by competitors that never do any estimation at all but are very good at pulling themselves out of tarpits of their own making.

This is a general argument against statistics. Or math, in general. Yes, dressing your bullshit in math can make people believe you more, but it doesn't change the fact that you're lying. Are we supposed to stop using math for good because evil people are using it for evil?

No, it's an argument against using statistics without first considering the strength of your data.

But being a good bayesian makes you do exactly this. The process of describing priors makes it obvious you need to do a sensitivity analysis to check how much the prior is influencing the conclusions...

> being a good bayesian

This is exactly what weirds people out about LessWrong folk. They talk about a tool as if it's a religion.

It's a running joke there.

The people need to get over it. LW crowd is a group of people studying a pretty specific set of subjects, focused around a single website. It's typical for such a group to develop their own jargon and insider jokes, which may look weird from outside. It's normal.

"Good Bayesian" in that context just means being an able user of Bayesian statistics, not necessarily holding any particular philosophical belief about what they mean.

Then you should take the Bayesian side, because Bayesians look at the data first, and they take their data as given rather than taking a null hypothesis as given. They don't just blindly go off and run a test (which assumes a particular prior implicitly that may be wildly inappropriate) and see what it says about the likelihood of their already observed data being generated by the test's assumed data generator.

How can you evaluate the strength of your data without using statistics? You've created a catch-22.

I'll speculate you have some sort of meta-heuristic and only apply this catch-22 under those circumstances? E.g. this catch-22 only applies to weird and socially disapproved topics?

On the other hand, one could argue that whenever the Church of Scientism sees someone using one of their favorite tools to argue in favor or a subject considered taboo, said church declares the use of said tool to be "invalid" or "out of scope".

I prefer to think of it in terms of the statistical inversion problem. That is, we have an event(s) that occur, which we may imperfectly understand. We take noisy measurements of that event. Clearly, the causal relationship is the events cause the measurements - a bad measurement does not cause the event to move.

But, in practice all we have are measurements, and from that we want to find an optimal (or good) estimate for what the events were. Hence, inversion.

Bayes formula expresses P(x|y) in terms of P(Y|x), so you can perform the inversion using bayes.

In many fields establishing the prior is difficult, hence frequentist methods are popular.

There are many techniques for the statistical inversion problem. Trying to track a ballistic object in a vacuum? Fit the measurements to a second order polynomial (parabola) and you are done (well, you have to decide least squares vs robust methods, but it is not such a hard problem in the scheme of things). Trying to track a manuevering jet, stock prices, or disease incidence rates. Now your model of the problem is much less clear.

We model lack of information as random variables. It isn't "random" in the deterministic sense, just in the sense that we don't know. Establish a good probabilistic description of that lack of knowledge in your prior, and you are probably going to get good result: this jet fighter is probabilisticly either moving straight, performing a coordinated turn, or performing an uncoordinated turn. Use a Markov chain to model those likelihoods (e.g.), and you may end up with good results. But if your modeling of the prior is poor, well, good luck to you, your output is probably nonsense.

Most measurements do affect the event.

ALL measurements at a quantum level at least!

Believe it or not, the Heisenberg Uncertainty Principle has nothing to do with measurement.

Is someone claiming otherwise?

Lots and lots of people, especially people who don't know about conjugate pairs or the Fourier transform.

I am only just learning about this stuff, but there are several things in this article that seem incorrectly explained. Conceptual clarity is paramount to me, so it drove me a little crazy!

> Bayes’ theorem is a method for calculating the validity of beliefs (hypotheses, claims, propositions) based on the best available evidence (observations, data, information).

Bayes theorem is a statement about probability, not "validity." This description makes it sound like Bayes theorem is a function BT(belief, evidence) = validity of belief. But it's not like that at all.

Probability is a way of measuring uncertainty. Things are uncertain for two main reasons: either we can't observe them directly ("do I have this disease or not?") or they haven't happened yet ("what side will this coin flip land on?"). (If you believe in a deterministic universe, the second is just a special case of the first.)

The "beliefs" (aka priors/posteriors) in Bayes theorem are statements of probability. To use the article's example, if it is claimed that 1% of the population has a certain disease, your "belief" or "prior" is that P(I have the disease) = 0.01. The article seems to get confused and think that the "belief" here is "I have the disease." Bayes theorem doesn't tell you about "the probability that a belief is true" like the article says, the belief is a probability. It also doesn't tell you if your belief is "valid."

Bayes theorem takes your existing belief about the probability of something and gives you a new probability that incorporates some evidence you observed.

Here's a proposal: Bayesian scientists shouldn't select their own prior. Instead publish how your results would update any prior, including the one picked by me, the reader.

I certainly haven't thought this through, but maybe this would make science more modular: combine the updates from M studies and calculate the new, combined update. Statisticians, does this work?

Yes - Laplace originally proposed this, it's a good approach (and incidentally the basis for Bayesian meta-analysis). Google "skeptical prior" for more.

I had a professor in graduate school who suggested exactly this - each study should conduct a meta-analysis of all previous studies on the subject, use that estimate as their prior, update, and publish for the next study...

The problem is, having tried it, this is much more difficult to do in practice.

This does happen. I've had paper reviewers request results with an appropriately selected uninformative prior: https://en.wikipedia.org/wiki/Jeffreys_prior

Typically, if you are practitioner in the field, it is not too difficult to identify instances where the result is highly dependent on the choice of prior.

On the second point. It's called meta-analysis and is an important area of research. However it's very far from operating automagically and requires substantial manual input.

On the first point this is equivalent to contracting a builder to build a house in any style.

Choosing a prior isn't always just about picking a wide band around 3 or a narrow band around 2. This is like the builder offering a choice of curtain colours from a swatch.

Making a commitment to implement any prior could be totally. Like the overcommitted builder being asked to reconstruct R'lyeh.

There was a lesswrong post on that back before they moved to lesswrong.com: http://www.overcomingbias.com/2009/02/share-likelihood-ratio...

The current fashion for BT really bugs me.

BT inverts conditional probabilities. If you can estimate P(E), P(H) and P(E|H) better than P(H|E) it will give you a better result. It is one of many probability identities. But someone it has become 'the one', as if, say P(H|E) = P(H&E)/P(E) isn't much use, but put two of those together: world changing.

I've seen so much crap come out of this fad. My particular favourite is in theology. William Lane Craig has demonstrated that Jesus raised from the dead, to a high probability. Richard Carrier has shown that there was no historical Jesus. Funny how few people ever run BT and find it contradicts their views.

I think part of the problem comes from a lack of understanding of the difference between frequentist and bayesian interpretations of probability. I've yet to see these folks show BT working in anything but frequentist data. And then they'll switch and use it to demonstrate why their Bayesian situation is correct.

Bayes does more than just invert the conditional. Of P(x), P(y), P(x|y), and P(y|x), if you know any three, then Bayes will give you the fourth.

It's just an equation. Garbage inputs will yield garbage outputs. In the realm of theology, it is of no more use than Pascal's Wager as an expected value calculation. All the input values are made up, so the output value is equally fabricated.

If you're using it on real, verifiable statistics, such as verified spam in an e-mail corpus, you can use Bayes to make a classifier to automatically identify spam to a high degree of accuracy. But if you are estimating for any of the three numbers you need to know, the fourth that you calculate will also be suspect.

Of course two people can come to different conclusions based on a Bayesian analysis of the same question, if their priors are different. The benefit of Bayes' Theorem is to make explicit the dependency of the result on the prior.

I just love how some guy, who by his own admission read a little Wikipedia on the topic, is critiquing a statistical method.

It would make for a much stronger argument if he actually showed some numbers where people are getting the priors wrong. That is, how often people get the priors wrong and the probability of mistake if they used a different strategy more commonly used in the field.

> The potential for Bayes abuse begins with P(B), your initial estimate of the probability of your belief, often called the “prior.”

tldr; priors matter

That was my thought as well. Garbage in = garbage out, that's pretty standard in most fields. I really didn't like how the author treated the theorem as if it's some sort of magic, aside from something everyone that's taken a college prob/stat class has derived from first principles.

The problem is that advocates do treat it as a magical thing. They extrapolate from the fact it is proven to the claim that all knowledge is Bayesian, to the implication that all Bayesian reasoning is knowledge.

This fashion is why, for example, BT has been used to both prove the resurrection of Jesus, and to prove that Jesus didn't exist: both to a very high probability.

I'm almost sad I've never met one of these people in the wild. I'd really like to sit down and watch someone, with a straight face, try to say they both have a set probability for their belief on Jesus AND the probability that some vague ~evidence~ would exist given that belief.

This was Dawkins' assessment in the use of Bayesian Probabilities to "prove" the god hypothesis...basically the numbers being entered were complete and utter fabrications, making the mathematics pointless except that they lent an air of quantitative rigor.

This headline does not reflect the article and is needlessly inflammatory. This article is an explanation of Bayes theorem and overall very positive of it. The "used wrongly" quote is just stating that it's not immune to biases and error. Pretty much any tool "used wrongly" can cause errors.

(In case it's changed, the headline currently reads "Bayes theorem used wrongly, can promote superstition and pseudoscience")

I think the title isn't quite so off. The article is trying to point out that people should be wary of using it wrongly.

Like other commentators have pointed out, the saying "garbage in garbage out" only helps when you stop to think whether or not you're putting garbage in.

They were doing great until they got up to the re-test, claiming that the second positive gives you 99% certainty you have cancer. That only works if the second test is completely independent from the first. If you repeat the first test a second time, only for those who get a positive result on the first, the same condition that caused a false positive on the first can cause a false positive on the second.

In reality, a cheap blood or urine test is likely to be followed by a more expensive test on a second portion of the same sample, then by an even more expensive tissue biopsy. Redoing an identical test only reduces random errors. It does not address the diagnostic bias of the test itself.

For instance, a pregnancy test detects hCG, from the placenta of a developing fetus. A man with various types of cancer or liver disease may also produce hCG, and can therefore produce a false positive for every test of that type, no matter how many repetitions. This does not give him greater confidence that he is pregnant!

Understanding Bayes also requires an understanding of event independence! For truly independent events, P(x|y) = P(x) and P(y|x) = P(y).

> Bayesians claim that their methods can help scientists overcome confirmation bias

The claim isn't that Bayesianism somehow prevents biases. Using a Bayesian approach is important in science because frequentism answers the wrong question[1].

    Many scientists operate as if the confidence interval is a Bayesian
    credible region, but it demonstrably is not ...

    I think the reason this mistake is so common is that in many simple
    cases ... the confidence interval and the credible region happen
    to coincide. Frequentism, in this case, correctly answers the question
    you ask, but only because of the happy accident that Bayesianism gives
    the same result for that problem.

[1] http://jakevdp.github.io/blog/2014/06/12/frequentism-and-bay...

(I'm referencing part 3 for the discussion of why frequentism is inappropriate in science, but I recommend reading the series from the beginning)

I feel like there are multiple questions getting conflated:

* Does Bayes help honest enquirers to find the truth? (Relative to what? At what skill level?)

* Does Bayes help bullshitters to hide the truth? (Relative to what? At what skill level of bullshitter and mark?)

Re the second, I think it's difficult to use Bayes to bullshit someone who understands Bayes as well as you do.

To me, the single most easy to understand form of Bayes' Theorem is:

    P(A|B) * P(B) = P(A∩B) = P(B|A) * P(A)

An intuitive explanation to the equation:

The (possibility of events A and B both happening) equals the (possibility of A happening when B is happening) * the (possibility of B happening), which equals to the (possibility of B happening when A is happening) * the (possibility of A happening).

Combining that with a Venn diagram:

    {A (A}∩{B) B}

Since P(B|A) means the chance of event B happening when A is happening, in this case which indicates that both A and B are happening, it's really just the area of (A∩B) divided by (A), which can be translated to (P(A|B) * P(B)) / P(A).

Kind of a clickbait title considering the article is a bit more nuanced than that.

I've had people seriously claim to me that using Bayes theorem to evaluate beliefs that one deals with in ones everyday life using evidence that one comes across in everyday life was likely a good idea and would reduce bias. I wish I'd had the presence of mind to point out that that did nothing to eliminate the selection bias of one's own experience. No mathematical formula can draw meaning out of weak or flawed evidence.

Trying to do so is like trying to 'enhance' a blurry photo so that you can see details in the photo that didn't exist.

Even if you only get weak of flawed evidence, then you do what you can to make the best decisions given that evidence. No one tries to suggest you do actual bayesian calculations on everything you know, for the simplest reason that it's not computationally viable.

But if your beliefs directly contradict bayes then you're doing something wrong - there's an inconsistency that's likely worth investigating, unless the matter is really minor. It's a sanity check for your decision making, not a constructive algorithm for the best decision.

> Cognitive scientists conjecture that our brains incorporate Bayesian algorithms as they perceive, deliberate, decide.

As a Cognitive scientist myself, this amused me. The reason being because in the 1950s, Cognitive scientists thought that the brain worked like telephone switching equipment.

Basically, we fit our current model of cognition to the most popular model of computing at the time. Looks like the trend hasn't stopped (although to be fair we we were talking about the Baysian model of cognition 20 years ago, so at least that one lasted a while).

Bayes in action: The author of this article had a set of prior beliefs about the Bayes's Theorem, which evolved after he reviewed more evidence.

Bayesian probability is catching on because the technology has finally caught up to the point where it's feasible. Thanks to the web and the proliferation of big data, we now have enough observations that Bayesian models can be trained (this was always the hard part). This doesn't mean we don't need smart people figuring out what things the model should and shouldn't look at; but it's at least possible to do today.

It also doesn't hurt that the Bayes theorem is at its heart a map-reduce problem. Where once Bayes' theorem was considered a cumbersome artifact for "brute forcing" probability, it's now likely faster than competing methods of statistical analysis.

We almost seem to be getting to the point in ad targeting where demographics are expressed in terms of a set of bayesian properties. You don't even care what the properties are; just that they're potentially more willing than average to use your product.

> It also doesn't hurt that the Bayes theorem is at its heart a map-reduce problem.

??? I'm not following. You could say that (numerical) integration is a map-reduce problem but that's pretty trivial.

Since we're on the subject, can anyone point me in the direction of how to account for correlated inputs? Without adjusting them, it can possibly give nonsensical probabilities (>1) but my math isn't good enough to decipher the few academic texts I've seen regarding this situation.

A (very simple) example: I trade stocks. My starting point is that I think a stock has a 50% chance of rising next year. Then I want to do a Bayesian iteration with the stock's P/E ratio based on historical data for stocks with similar P/E ratios. Then I want to also incorporate the P/E ratio of the industry the stock is in. Obviously these two inputs are correlated and if you have enough correlated variables, the whole thing breaks down because the simple theorem only works if all the inputs are independent of each other.

Same topic but differently presented by Quanta Magazine https://www.quantamagazine.org/20151216-physicists-and-philo...

The fundamental insight of bayesian reasoning is that prior probabilities can matter a lot in some situations, even when the evidence seems fairly strong.

Look at scientific papers. Usually there is something called a p-value attached to it, which is usually around 0.05. This means there is only a 5% chance that the same result could have been produced by random chance.

How strong is that evidence, really? What if you started with a prior probability of 1 in 1,000 that it's actually true. That is, before you saw the paper, you would estimate there's only a 0.001% chance it's true. Out of 1,000 similar studies, 49 will be false and yet return a <0.05 p value. Only 1 will publish actually true results.

If you have some time to spare, here's a great paper on a similar topic: the (mis)use of mathematics and in particular equilibrium methods in economics after Samuelson and Hicks ("What Went Wrong with Economics?", Peter J. Boettke, 1997): http://www.the-dissident.com/Boettke_CR.pdf

Pretty much every debate on economic policy between the left and right is full of fallacies emanating from this episode in history.

I find Bayesian subjective probability even more interesting. It has been successfully applied in unique situations, such as during the cold war, by a panel of "experts" in the US, to locate Russian test fired rockets in the ocean.

modal logic is an equally valid way to get at a lot of the quandaries associated with bayesian ideas.

bayes' thm by itself is totally not a "big deal." the idea that every probability has an associated prior, even if it's not explicitly written in the notation (so think "prior of prior") is an interesting attempt to cope with uncertainty in a rigorous way.

i do agree that in some places in the sciences, while "the numbers don't lie," the stats can be misleading. still, i can understand why it's useful to make some statements with statistics in order to quickly arrive at some first-order approximations.

Yudkowsky's 'Less Wrong' group of 'rationality' followers, aimed to try to force-fit all of science into the Bayesian framework. And of course it doesn't work at all.

I think Andrew Gelman's criticisms are right on the mark.

Where it breaks down is when you try to apply it to reasoning about your own internal thought processes. It was never intended to do this. As Gelman correctly points out, it is simply invalid to try to assign probabilities to mathematical statements or theories, for instance.

You see 'Less Wrong' followers wasting years of their lives engaging in the most unbelievable and ludicrous intellectual contortions to try to force-fit all of science into Bayesianism.

Go to the 'Less Wrong' blog and you can read reams and reams of these massively complicated and contorted ideas, including such hilarious nonsense as 'Updateless decision theory' and 'Timeless decision theory'.

---

In terms of dealing with internal models or hypothesis, I think the correct solution is not to assign probabilities, but rather to assign a 'conceptual coherence' value, so for instance rather than say 'outcome x has probability y' (where x is a hypothesis) you should say 'concept x has conceptual coherence value y'

Conceptual coherence is the degree with which a hypothesis is integrated with the rest of your world-model, and I think it accurately captures in mathematical terms the ideas that Deutsch was trying express.

Probabilities should be viewed as just special cases of conceptual coherence (in the cases of outcomes where you are dealing with external observations or sensory data, Bayesianism is perfectly valid).

Then all of the problems with probability go away, and none of the massively complicated theories expounded on 'Less Wrong' are necessary ;)

I think Stephen Bond did some excellent takedowns of the identity politics that has arisen around Bayes' Theorem back in the day. I wonder where he's at these days.

The Cult of Bayes' Theorem

http://laurencetennant.com/bonds/cultofbayes.html

> One of Yudkowsky's constant refrains, appropriating language from Frank Herbert's Dune, is "Politics is the Mind-killer". Under this rallying cry, Lesswrong insiders attempt to purge discussions of any political opinions they disagree with. They strive to convince themselves and their followers that they are dealing in questions of pure, refined "rationality" with no political content. However, the version of "rationality" they preach is expressly politicised.

I've seen this type of writing before. It's a kind of twisted pseudo-criticism you write against a group you dislike. You can compose stuff like this against any group. It sounds believable from the outside, especially if you start sceptical to begin with. But take a closer look - it's actually full of ad-hominems, cherry-picking facts and presenting them in worst light possible. I've been a part of several groups that were targeted by such prose - first the religious group I grew up in, that is a minority in my country; then the school I went to. My university year used (a very lite version of) such criticism against another, so I've seen it from the other side as well. Hell, people write shit like this about HN!

It's hard to defend against such criticism. You'll get boggled down in refuting specific accusations, but this is something you can never win. The only winning move is to ignore it completely. Personally, I shun and shame people who write such stuff, regardless of whether I agree or disagree with their victims. Dishonesty is a poison that destroys societies.

TL;DR: this text is harmful, malicious bullshit. If it at least offended people with style, there would be something to save it.

I know you're a huge advocate for Lesswrong, but not "everybody" or "anyone" has quotes ripe for picking like Yudkowsky. Stephen Bond is not just throwing some opinion out there, he's backing it up with first-hand sources:

Yudkowsky on his simplified views of why race gets brought up:

> "Race adds extra controversy to everything; in that sense, it's obvious what difference skin colour makes politically".

> "Group injustice has no existence apart from injustice to individuals. It's individuals who have brains to experience suffering. It's individuals who deserve, and often don't get, a fair chance at life. [...] Skin colour has nothing to do with it, nothing at all."

Yudkowsky on our current societal structure, adulating the people who give him funding:

> One of the major surprises I received when I moved out of childhood into the real world, was the degree to which the world is stratified by genuine competence.

Yudkowsky writing short stories about a society where rape is legal, leaving himself ample room for plausible deniability, but putting it up "for debate":

>> "No, us. The ones who remembered the ancient world. Back then we still had our hands on a large share of the capital and tremendous influence in the grant committees. When our children legalized rape, we thought that the Future had gone wrong."

>> Akon's mouth hung open. "You were that prude?"

(https://web.archive.org/web/20131206200429/http://lesswrong....)

I think some well-meaning LessWrongers get caught in the crossfire, but I think the essay makes a very well grounded case for the blindspots "rationalists" have towards politics that suit the identity of people like Yudkowski.

I'm not that huge advocate of LW. I used to frequent the site, I read the Sequences and generally liked what I saw.

RE parts you quoted. Of course they look bad, because they've been ripped out from context.

Race:

http://lesswrong.com/lw/kk/why_are_individual_iq_differences...

This is an article asking, to quote: "But why is it that the rest of the world seems to think that individual genetic differences are okay, whereas racial genetic differences in intelligence are not?". Yudkowsky seems to argue that making big controversies around whether there are, or are not, differences between "races" is missing the point; it's just one of many variables and we should focus on fixing intelligence disparities for everyone.

Society:

http://lesswrong.com/lw/ub/competent_elites/

This blogpost is marked as controversial from the get-go. It covers a quite interesting theory IMO - that the very common meme, which says that elites are stupid and evil, is in fact wrong. Eliezer argues towards a more socially uncomfortable opinion - that elites are, in fact, smarter and better at organizing things. Given that this is a belief most people are heavily biased against, it actually may be the "thing you can't say", per PG's essay at http://www.paulgraham.com/say.html.

Story:

Geez. This is a sci-fi story. Moreover, it's explicitly designed to fuck with reader's moral intuitions. That's its entire point. Personally, I find it fun and insightful, but it is heavy. You can read the whole work here:

http://lesswrong.com/lw/y4/three_worlds_collide_08/

And to be clear. I'm not idolizing Eliezer. He's just a man who thought a lot about some stuff, and wanted to share it. He sometimes gets it wrong (and, as opposed to many, has at least the courage to admit he was wrong). But I absolutely hate this kind of bullshit pseudocriticism when it's directed against anyone - be it my friend or enemy, be it someone I admire or despise of. Eliezer is not beyond criticism, but we can do better than that.

I guess I just disagree with you that they're taken out of context?

1. Bond posits Yudkowski thinks that racism is mostly due to genetic difference, and not about the deliberate, mostly political disenfranchisement of minorities.

You say this is a wrong because his appraisal of the situation as being caused by genetic disparities is benevolent, and that he wants to work to "fix" it.

These aren't in contradiction- to say that it's unfortunate that society is racialized, but that it's due to traits rather than politics, is better than being wantonly racist, but still a well-known form of fallacious racism. Some would say this latter one is actually worse in terms of perpetuating the situation.

2. Of course it's "controversial from the get go". He looks at the disproportionate representation of certain people in ie: tech, and concludes that it's because they're "more competent". Nothing here seems to be a counter, you're just basically saying "it's politically incorrect so it's probably right".

3. Clearly it's just a story. This point eludes exactly nobody. The whole point is that these kind of people find these questions ("is rape really bad?") really intriguing rather than obvious. It makes you wonder about the power of Bayesian reasoning- the exact point of the essay.

I think you're too uncharitable to critical writers like Bond (it took me a while to acquire a taste for this vicious kind of writing), underestimating their and their audiences' understanding. As a result, you think this context adds way more than it does.

The racism and feminism points Bond makes sound to me like typical social justice warrior bullshit - no matter what you do, you're a misogynist racist. Disagreeing with the modern trend of mindless outrage paints you as an enemy. So does ignoring the topic. So let me skip it, because I do disagree with Bond very seriously about that part of the text, and I don't want to start another SJW subthread here.

> Nothing here seems to be a counter, you're just basically saying "it's politically incorrect so it's probably right".

No; I'm saying that just because he's politically incorrect, doesn't mean he's wrong. And it most definitely does not mean Bond gets to assume all the bad things he did about the author. Eliezer is just toying with an idea based on some observations. Personally, I find his idea intriguing and worth considering, given that time and again I've learned that when the general population seems to believe something counterintuitive and self-serving, like "those rich people are stupid and evil", they're usually wrong.

> The whole point is that these kind of people find these questions ("is rape really bad?") really intriguing rather than obvious.

I think we disagree on the interpretation. Many stories raise weird questions about morality; that's the reason we have literature. Please find me a place on LW where people, and Eliezer in particular, were seriously advocating for rape. Also it's worth noting that the rape angle was just a passing remark in the story, a way to shake the audience a little bit - and otherwise in no way relevant to the plot.

> I think you're too uncharitable to critical writers like Bond (it took me a while to acquire a taste for this vicious kind of writing), underestimating their and their audiences' understanding. As a result, you think this context adds way more than it does.

Maybe I am, but it's because I've been a member of groups that were on the receiving end of such writings, and spent a lot of time debunking them point-by-point to concerned friends and colleagues who stumbled upon them. It's painful, and you realize just how helpless you are against people who argue dishonestly.

"1. Bond posits Yudkowski thinks that racism is mostly due to genetic difference, and not about the deliberate, mostly political disenfranchisement of minorities."

You don't need to take the author's word for it. Look at the article itself.

The article is not about the question of whether race affects intelligence. It's saying that that question is much less important than the fact that individuals have different IQ levels regardless of race. At least in the context of this article, which is asking whether "God is fair".

To make it crystal clear - Yudkowsky is saying:

1. Forget race. There are clear, obvious, and gigantic genetic differences in individuals which cause differences in intelligence.

2. This is "unjust".

3. We should be upset about this, and try to fix it.

That's all he's saying in that article - it is not at all racist.

If you think otherwise, please - quote the relevant part of the article and explain yourself.

I'm not gonna do a good job at explaining this, because it's far from my subject area, but the problem is that your demand for "the relevant part" asks for blunt evidence of a subtle phenomenon.

When we talk about politics, context matters. Someone like Yudkowsky doesn't believe stupidly racist things, like a KKK member or turn-of-century politician, or even modern anti-immigration people. Bond's targeting something much more subtle.

The point is that here you have a man who naively keeps trying to push the dialogue "beyond race" ("forget race"), in a forum where if you scroll down to the comment section, you'll see Jeff H. with 5 upvotes defending Watson's racist remarks, with Epiphany at 0 upvotes talking about the cultural reasons why IQ tests fail.

It's about the framework, and what it allows, and about what allows the people championing it to be naive or indifferent about what it allows. Someone discussing coldly and clinically the pros and cons of rape isn't a rapist, but there's something else about someone who can have discussion coldly and clinically. Some people take pride in their level-headedness about tough topics, others would take it as a signal of their lack of empathy with the victims, and the signal this sends to people who do feel passionately about it.

And so Yudkowsky will discuss race in this goofy aloof kind of way- "maybe black people are intellectually inferior, let's talk about it, does it even matter in the end?", and some people will believe that this is the right kind of coolness to produce neat solutions, and others will believe it robs the steam off of the emotional connection that would engage people in political action for change.

You have to suss this kind of thing out, and if you plain don't want to believe me that there's something enabling towards racism and sexism and other various supremacist ideologies in the LessWrong ecosystem, you don't really have to.

There is definitely something enabling about it. A lot of modern politics wants you to NOT believe black people are stupid criminals.

Yudkowsky would almost certainly endorse "if black people are stupid criminals I want to believe black people are stupid criminals, if black people are not stupid criminals I want to believe they are not stupid criminals". (This is the litany of tarski, which he endorses, applied to a specific case.) One could certainly view this as "enabling" supremacist views under the circumstances that reality supports them.

Is a fair summary of your critique that Yudkowsky is rationally and unemotionally reasoning about problems which are normally addressed via appeals to emotion, and this subverts emotional appeals to the masses to take certain actions?

I've read your post several times, and this is the most charitable takeaway I can come up with. But this sounds insane so I feel like I must be misunderstanding something.

Someone who feels injustice, but hasn't had the time or resources to codify it into a calm rational treatise is said to be speaking from emotion. Someone who merely experiences base greed, but is able to justify it to themselves as righteous and deliberate and socially beneficial with a bunch of babble, is considered rational.

And so, the idea that the rigid set of discursive boundaries that LessWrongians impose upon themselves may favor a political outcome (the status quo) seems ridiculous to you, because you use "rational" as a compliment and "emotional" as an insult.

Can you find me an influential LessWrongian who describes making emotional decisions based on greed as "rational"?

Based on my reading, a profit seeker obeying his gut and losing money is irrational. In contrast, a social justice type observing that assortative mating causes inequality and therefore attempting to reduce educated women's marriage choices is rational. But maybe you've read more than me, and can figure out what I missed?

And so, the idea that the rigid set of discursive boundaries that LessWrongians impose upon themselves may favor a political outcome (the status quo)...

This is a completely new critique of the Less Wrong crowd. The normal critique, and the specific one you cited, claim that Less Wrong is too far from the status quo and concerns itself with things like the singularity.

Adding to my confusion is that in the very post you criticize, Yudkowsky explicitly advocates fixing intelligence deficits with "sufficiently advanced technology, biotech or nanotech". How is fixing every person with an IQ < 150 using nanotechnology remotely preserving the status quo?

(Note also that Yudkowsky is explicitly advocating that if genius isn't uniformly distributed, as social justice types claim to believe, we should explicitly change the world to make it so.)

I can't seem to post a reply, so I'll wrap it up here.

I'm criticizing an ecosystem. Yudkowsky-types noodle with weird hypotheticals, others with elitist views get validation. Fantasizing of fixing our current issues with futuristic tech, using it as a yard-stick to criticize ie: a collective black identity from forming a political block, is not explicitly pro-status quo but ends up being so in practice.

So the only criticism here is that views of various people on LW forum do not conform to the mainstream social justice outrage narrative, where everything needs to be politicized and "status quo" must be fought?

That's sort of the entire point of "Politics is the Mind-Killer" statement, a point which Bond also missed - LW community wants to focus on effective ways to deal with actual problems, as opposed to doing politics. They're not criticizing "a collective black identity", speaking against them "forming a political block". They're not talking about it. It's besides the point of that article and mostly besides the point of the entire community, which tends to focus on how to make things better for everyone.

Frankly, I find it funny to see accusations of racism aimed at people who are known to seriously, and not just as a figure of speech, consider humanity as one great family who are in it together. But then again, everyone of us who is not outraged is secretly a racist and supports the enemy.

Also consider the core statement of the Mind-Killer article:

"Politics is an extension of war by other means. Arguments are soldiers. Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back—providing aid and comfort to the enemy. People who would be level-headed about evenhandedly weighing all sides of an issue in their professional life as scientists, can suddenly turn into slogan-chanting zombies when there's a Blue or Green position on an issue."

It's perfectly OK to avoid that kind of political discussions. I'd say it's weird to actually partake in them.

I'd love to reply to you all (ie: siblings) but this discussion is just sprawling (not a fan of "tree-and-leaf" fanning arguments, I much prefer linear forums), and I just can't put in the time.

FWIW, you strike me as a good person, as do many LWers. I wish I could communicate to you the nuance of my issues with statements like "humanity as one great family", but I'm a newbie at the study of ideology myself, so I wouldn't do a good job.

À la prochaine!

I'm not very familiar with the subject matter, but do you realize that this concluding argument is very weak?

Firstly, I don't think I've seen mentioned an instance where Yudkowsky seems to be trying to prevent a collective black identity from forming a political block.

Secondly, you've suggested that the mechanism by which Yudkowsky's material end up as bullets for people who desire to perpetuate prejudice is where the prejudiced party misrepresents Yudkowsky (thereafter, Y). Therefore, your reasoning goes, Y and the LW commenters are guilty for engaging in subject matter that are ripe for appropriation.

Unless I'm mistaken, your accusation is one that is pretty unjust in itself. You are accusing Y with a moral crime for Y&LW's association with (racially) prejudiced groups that have misrepresented Y as prejudiced. This is despite that Y has neither affirmed this association nor mentioned anything prejudiced, his crime being entertaining "weird hypotheticals". Rather, if such a thing has occurred, isn't Y a victim of the prejudiced groups himself?

I think your premises are mistaken on several fronts, I'm annoyed by your (mis)characterizations of LW and Eliezer, and I disagree strongly with your conclusion...

but your comment helped me actually understand this position for the first time, so thank you for articulating it.

>The whole point is that these kind of people find these questions ("is rape really bad?") really intriguing rather than obvious.

That isn't the question the story is asking. Second of all, Eliezer says:

>This is a work of fiction. In real life, continuing to attempt to have sex with someone after they say 'no' and before they say 'yes', whether or not they offer forceful resistance and whether or not any visible injury occurs, is (in the USA) defined as rape and considered a federal felony. I agree with and support that this is the correct place for society to draw the line. Some people have worked out a safeword system in which they explicitly and verbally agree, with each other or on a signed form, that 'no' doesn't mean stop but e.g. 'red' or 'safeword' does mean stop. I agree with and support this as carving out a safe exception whose existence does not endanger innocent bystanders. If either of these statements come to you as a surprise then you should look stuff up. Thank you and remember, your safeword should be at least 10 characters and contain a mixture of letters and numbers. We now return you to your regularly scheduled reading. Yours, the author.

The point that the protagonists legalized rape, is to demonstrate that the protagonists morals are just as far from us, as the baby eaters or super happies. That the story isn't about two alien societies, but three.

Second of all, what "these kind of people" find intriguing is that our morals are for the most part due to our circumstances. That in the future, people may very well accept what we find today to be abhorrent, and find abhorrent what we accept.

That is the point of "is rape bad?". Because it is simply an example of something that is obviously bad. Just like some time ago there were other things that were considered obviously bad, that today we accept (same sex relationships for example). And if we are aware that the future will look back and mock us for our obviously wrong morals, then maybe we will be faster to get on the ball with the next gay rights or what ever.

Are there moral beliefs you hold today, that you think the future might condemn you for? Or do you think in this time and place you've acquired a perfect set of morals?

These sorts of quotes can reveal a lot about the source that quoted them, if you track them back to their original source and check for reasonableness there. I think you'll find that each of these quotes had its meaning or connotation dramatically altered by the removal of context.

Those parentheses obscure a significant point. The full paragraph is

>So, in defiance of this psychological difference, and in defiance of politics, let me point out that a group injustice has no existence apart from injustice to individuals. It's individuals who have brains to experience suffering. It's individuals who deserve, and often don't get, a fair chance at life. If God has not given intelligence in equal measure to all his children, God stands convicted of a crime against humanity, period. Skin colour has nothing to do with it, nothing at all.

He's claiming that skin color has nothing to do with the (according to him) injustice of different people having different intelligences. He's not claiming that skin color has nothing to do with why people don't get a fair chance at life. "Skin colour has nothing to do with it" is making a moral claim, not an empirical one.

If your ideas can stand on their own merit, you can defend them against criticism without resorting down to saying things like "oh it's a pseudo criticism and cherry picking facts". You're the one calling it bullshit. You even admit you wouldn't want to refute specific accusations.

And then go on to call it "malicious bullshit".

So, basically what you're saying is that we shouldn't listen to this criticism (which provides very valid points btw) but listen to you, when all you've done is attack the critique in a classic ad-hominem fallacy.

See the problem here?

Let's try: as far as I know, this bit: Under this rallying cry, Lesswrong insiders attempt to purge discussions of any political opinions they disagree with.

Having hanged around LessWrong for quite some time, I'm pretty sure this point is false. How am I to show it however? Dump the whole site to show the absence of such purges? And how the inevitable counter-example? Quite a lot has been said there, we're bound to find unacceptable behaviour somewhere. All I can provide is a testimony. In other word, anecdotal evidence, which we all know to be weak.

There's another way to say "pseudo criticism and cherry picking facts". It's the fully general counter-argument. If the criticism can indeed apply to all groups, then it effectively applies to none, because it doesn't provide a way to discriminate different groups.

Alright, lets see

> Having hanged around LessWrong for quite some time, I'm pretty sure this point is false. How am I to show it however? Dump the whole site to show the absence of such purges?

To refute the above statement, only one statement to the contrary would be a good start - it would provide me with enough evidence to consider your side of the story and think that perhaps the critique isn't completely fair.

> Quite a lot has been said there, we're bound to find unacceptable behaviour somewhere.

This can be said for any popular community. Does that mean we should not critique them? Or that some how invalidate any criticism against them? No.

The thing you're missing here is not the number of counter-examples, but a general cultural trend. HN, for example, is a tech oriented community, but political discussion isn't anathema here. I can show this by reading a few posts and noticing how people are comfortable discussing differentiating political opinions here. It's not perfect argument, but it's a good start, and goes a long way towards making me appreciate the environment.

Here, there is an important role played by so called "leaders" in community and how tolerant they are as well, by the way. And I know for a fact that Elizer isn't one of them. Then again, what else do you expect from a guy who tries to edit his own Wikipedia page.

> If the criticism can indeed apply to all groups, then it effectively applies to none, because it doesn't provide a way to discriminate different groups.

I'd happily accept this if you specifically mention which criticism in the parent link can be applied to all. The critique is very, very specific. It starts by defining what Bayes Theorem is, what LW and their ilk think about it, more of their pseudo-intellectual talk, the cult of personality, the AI apocalypse doomsday scenarios, etc etc.

You want me to read it? Okay.

I don't like the introduction, mostly because I believe what he ridicules to a degree (AI is an existential risk, I have enjoyed HPMOR more than Rowling's work, and I'm not quite sure that making lots of money and give it away is worse than working directly for whatever cause we want to support).

Bayesian grace

Bayes T-shirt means certain asshole. Well at least it made me smile.

I agree that Bayes' theorem is not that notable. The product rule, more fundamental, is more representative of the laws of probability (the actual basis for bayesianism).

Of course, the "formula for the perfect brain" is computationally intractable…

The association between Bayesianism and Neo-Liberalism looks like an ad-hominem attack —doesn't apply to me, at least.

I can't comment on this "Bayesian revolution". But I'm already suspicious of the whole essay at this point, and cannot trust this paragraph.

Amazing Bayes

What the author fails to acknowledge here is that to the extent it can apply, Probability theory is that amazing. There are, like proofs of it working very broadly, and it relies on very few axioms. (Jayne's work in Probability Theory: the Logic of Science leaves little doubt about that.)

Absence of evidence is evidence of absence. Often very weak evidence, but evidence nonetheless. Whoever believe otherwise doesn't understand probability theory. (Maybe the author conflated "evidence" and "proof", or "evidence" and "strong evidence"?)

The correct application of probability theory is computationally intractable in many cases. I can see how it would be unworkable for historians, who have to juggle with many many kinds of evidence. Not having read Richard Carrier however, I'm not sure this objection is not yet another strawman.

Accusing Bayesianism to be responsible for confirmation bias is ridiculous however. Confirmation bias does not follow probability theory. It often follows an incorrect application of it however.

Less Wrong

Okay, I'm out. The insults are too blunt, came too soon, without any justification so far. SIAI (now MIRI) as a doomsday cult is a strawman. As for LessWrong, Eliezer specifically warned about the dangers of using rationalist's tools for rhetoric purposes —how your own biases can increase when you know about biases.

I'm not taking the effort required to search for and debunk any justification that might come later. He just used up my patience.

Yes, I see the problem you mention. All I offer is, as a person who spent some time hanging around that group but not exactly an insider, my honest assessment: this criticism is very unfair, very hurtful, and mostly bullshit. You don't have to believe me - but you now have a second data point for your consideration.

I did engage with some points in comments below though, to show examples of this cherry-picking facts, ripping them out of context and twisting them into supporting something completely untrue.

But my general point is that this kind of criticism is impossible to defend against. To prove that it's mostly bullshit would take me literally writing a book - in which I would surely make mistakes, that then could be used to discredit it entirely in a single article. This is the asymmetry of dishonest arguing - you can get your results with 1/100 the effort if you're willing to deceive your reader.

When people read a powerful criticism "debunking" things about a group they vaguely know, the default for most is to agree with the criticism. For some reason it's natural to humans. All I want now is to make people stop for at least a moment, consider that the article may be unfair, and to not make judgments before double-checking the claims.

Maybe I could've written it in a more dispassionate way. It's hard for me - not because I like LW, but because I've been in other groups targeted by this kind of criticism, spent way too much time trying to defend the group from it, and I know the crushing feeling that no matter what the truth is, it'll be twisted and bent until it can be used as a weapon against you.

> this criticism is very unfair, very hurtful, and mostly bullshit. You don't have to believe me - but you now have a second data point for your consideration.

No I do not. All you've provided me is a point of view without anything resembling rational reasoning. I believe that trains can fly. You don't have to believe me, but now you have another data point for your consideration.

> this kind of criticism is impossible to defend against.

See my initial statement. All you have to do is point flaws in the above link.

> To prove that it's mostly bullshit would take me literally writing a book - in which I would surely make mistakes,

If you're not confident in your ability to assess and refute an argument, then why bother making it?

> that then could be used to discredit it entirely in a single article.

Nobody is saying the article above is perfectly written, or that the critique is 100% on point, but that does not mean the broad point he makes has no merit. Same would be true for the hypothetical book you write.

> When people read a powerful criticism "debunking" things about a group they vaguely know, the default for most is to agree with the criticism. For some reason it's natural to humans.

Do you have any basis on which you're making that claim? No, because if you did, you'd provide that.

> All I want now is to make people stop for at least a moment, consider that the article may be unfair, and to not make judgments before double-checking the claims.

But WHY? WHY? You've, objectively, given me ZERO reason besides "it's malicious and people are biased when reading debunking articles". You've clearly refused to engage in the content. You haven't even quoted a single line and proved that it's false. Whereas the article in question is an in-depth exploration of the community of people, their beliefs, the personalities involved, what they believe, their education background and many more things. What have you done? Nada.

I can go on and tear down your whole reply, but I'll stop here.I come off with the impression that you're taking this personally and therefore, as a natural response, you're defending it, which is fine. I'm not trying to be a dick, but perhaps you should work on your argumentation and logic skills before trying to defend or take any side.

You're reading me very uncharitably. I believe that debunking this article properly, point-by-point and with overwhelming evidence, will take stupidly big amounts of time and is almost an impossible task - since no matter what I say, people with the same approach as the author can find something else to twist into criticism. On the other hand, Googling quotes he uses and checking them with source material is easy and provides ample proof that the article is dishonest. It's something a reader can do, and I want to clearly state that in this case, they absolutely should.

RodericDay asked about three concrete examples taken from the article; each of them is a perfect example how utterly nonsense this article is. I provided relevant source material and explained the real meaning of quotes Bond uses when taken in context of said source material. So did other HNers. Check out the responses in the parallel subthreads. I could do it for every one of his sentences but frankly, I feel it's a waste of time. I think I've proven enough that this is not an honest criticism, and fact-checking the rest of that article shall be the task for the reader, who is now properly warned.

This was posted on the Less Wrong Facebook page recently, where the reaction was mostly "There's no substantive criticism in this article, only mudslinging".

Other reactions include "I was raised in an actual cult, and the differences I see include that LW encourages debate and EY is frequently criticized."

Some commenters did agree with at least parts of the article, though. Even the ones that agreed said it seemed to be more "personal attacks" than "excellent takedowns", though.

I personally think it seems kind of... "smug", I think, is the word I want to use? "Smug" writing isn't necessarily wrong, but it generally seems to have other priorities than discovering the truth. I disagree that I'd eventually acquire a taste for this "vicious" kind of writing - I've read smug writing that I otherwise agree with, and it still makes me uncomfortable.

I don't want this to get too meta, but I like the style of your post. If you're bored, could you tell me what you think about this article?

http://gawker.com/on-smarm-1476594977

That is a very long article...

The point seems to be that while there's been a lot of criticism of snark (too much negativity), smarm (too much positivity) is also bad.

It seems to me that the way he describes them, snark and smarm are pretty similar: snark is being smug (acting unnecessarily superior) about someone else doing something wrong in judgment (like being ignorant or making a bad decision), while smarm is being smug about someone else doing something morally wrong.

I would probably mostly agree with the idea that smarm is bad, although I'm not sure I would have spent 30 pages on it.

He defends snark as criticism of smarm, but I think it's important to note that while criticism or negativity itself isn't necessarily wrong, it's the smugness that makes it bad. I'd probably describe smugness as criticism mainly for looking good to your ingroup, and for making your target feel bad.

For instance, I can imagine a good teacher criticizing me, but I can't imagine a good teacher being smug about it. Being smug seems correlated with impure motives, which decreases the accuracy and diminishes the trustworthiness of the criticism, to me.

The article is less about opting for snark over smarm, than it is about reading too much onto the method of delivery ie: "I can't imagine a good teacher being smug about it", being politicized in a particular way.

It kinda runs counter to the position you're advancing, but I thought you'd find it interesting anyway.

Fair enough. I started skimming around halfway through so I missed that.

I'd agree that there are limits to what you can read into the method of delivery, but:

I think anger and indignance are valid attitudes to have, and that seems largely what he's defending. They're not like smugness in that I can still imagine someone sincerely caring about the truth having them.

I'd also point out that I react with discomfort when I see smugness from my side, and annoyance when I see smugness from the other side, and these are physiological reactions that can't exactly be controlled.

I guess I didn't see much actual content in the Bond article (ironically, most of _that_ article was criticizing method of delivery), so the method of delivery was all I had to comment on.

This is written by a person who doesn't know what he's talking about. Any time people diss Bayesian stats and name Yudkowsky in their argument (a controversial guy without a high school degree), they haven't got a clue what they are saying.

Bayesian stats is a solid mathematical theory. It is used everywhere in academia and industry, is the foundation of a lot of AI & Machine Learning research, powers many technologies that, among other things, allow you to locate this thread on the Internet and post that ignorant link here.

Please stop promoting BS.

This is why people don't take philosophers seriously: They apparently can only attack strawmen, and badly at that, using emotion in place of reasoned arguments.

http://elis.dvo.ru/~lab_11/bib/hershkowitz-nadal.pdf I found a thesis on the topic.

Seems the upper bound of estimation when using bayesian probability for estimating parameters with data gets overestimated when the size of the data grows.

It does not makes sense to me, can someone explain this to me as if I was an idiot?

PS one of the searcher in his lab used to say: if you try to find to hard a physical law, you might eventually find it.

(quote from the thesis below)

Most of the results concerning the behavior of the mutual information, observed for this particular family, are ‘‘universal,’’ in that they will be qualitatively the same for any problem that can be formulated as either a parameter estimation task or a neural coding task. .... Besides the asymptotic regime p large, N arbitrary, we have also considered the case of large N at any given value of a5p/N. In this regime we have both replica calculations and exact bounds, in particular, an upper bound for the class information and explicit upper and lower bounds for the mutual information obtained with the techniques of [7]. The results suggest that the replica symmetry ansatz give the correct solution. The lower bound is then quite good whereas the upper bound overestimate the mutual information by a factor that keeps increasing with the data size.

So let me get this straight, statistics can be misused? I almost stopped reading as soon as I saw the image from the big bang theory. (That show presents nerds the way other people want to see them. Not the way nerds actually are.) It appears that the author of this article is just a journalist:

http://www.johnhorgan.org/

I don't see anything in there about him ever being a scientist, a statistician, or anyone who would actually use this theorem. In fact, he even went to school for journalism.

I really don't understand where this author is coming from when he says things like, "I conveniently decided that Bayes was a passing fad." Why should we care about his opinion on the matter? Is he reporting to us what Bay Theorem is and its significance, or is he giving us his uniformed opinion on the matter?

Reply on news.ycombinator.com