Poor Man's Scientific Method | Hacker News Reader

Poor Man's Scientific Method

250bpm.com

52 pointsrumcajz40 comments

I was happy to find this, because I've had a number of similar debates - some with very intelligent people! My appeals to the mechanism of action went over everyone's head and convinced no one. Citing studies got us lost in the weeds as they asked questions about the study that I didn't know the answer to. (Who paid for it? Did they adjust for ___? Did they try ___?)

I agree with the problem, but I don't agree with several points. I'm also not sure about the solution.

> "Today, it's common that a licensed doctor prescribes a homeopathic medicine, which is scientifically known to be ineffective."

Are there really? How common is "common"? This is the first I'm hearing of it, and I'd like more than the author's word. Perhaps a study? :)

>"the profession is regulated by the state and the doctor obtains his license only after finishing a school (where he is taught scientific method along with any specific knowledge needed in his field of expertise), having some actual hands-on experience and taking the Hippocratic oath."

This is actually one of the problems - our culture's over-reliance and expansive trust in state-mandated licensing. The state is simply not qualified to say who can do what or whether they're actually competent in a field, especially once we start talking about specialties.

Because the state lacks the time and experienced professionals in a given field, we came up with certification boards. There are some great boards made up of real-life specialty experts who certify doctors as being competent in their little niche.

Unfortunately, this gets us into the same problem: how do I know which boards I can trust? I guarantee there are homeopathy boards that certify doctors. Looking into the legitimacy of a board is just as hard as looking into the legitimacy of a paper.

The answer has to be education - anything you can do, the "alternative science" crowd and profiteers can do, too. Unfortunately, neither I nor the essay's author seem to have a great cure-all answer for people who don't have the time, interest, or intellect to educate themselves.

"How common is "common"? This is the first I'm hearing of it, and I'd like more than the author's word."

They're scientifically proven to be medically ineffective although for people with psychological non medical issues they are scientifically proven to reduce anxiety. Daughter's got a cough. Moms freaking out. Doc verifies its not bacterial so there's nothing to do but chill out, these viruses take care of themselves, rest and hydrate. Moms freaking out. You treat MOM not the daughter with a harmless magic potion. Mom is now calm, that makes daughter calm and unstressed, and thats medically proven to help or at least not harm someone who needs hydration and rest, and issuing poorly distilled water is medically proven to be much safer for everyone involved than giving antibiotics for a viral infection.

I'm a pretty strong supporter of homeopathic treatment of viral illnesses, at least in comparison to the US (edited: informal) standard of antibiotic treatment. Its enormously safer than antibiotics and its just as effective WRT virus infections. Just make sure its really viral in nature not just a poorly tested culture test in the lab.

(edited to explain I'm not talking theory here; my daughter's pediatrician has recently switched from antibiotic treatment of minor viral infections like coughs and colds to homeopathic treatment... when I found out from my wife what my daughter was prescribed, I was momentarily furious, come on lady I'm a hard science guy don't BS me with homeopathic stuff like that, but I thought about it for about five minutes and decided to high-five her next time I see her, it really is genius in that people too dumb to understand antibiotics have no effect on viruses are too dumb to know homeopathic stuff doesn't work, so the net result is positive)

I've been trying to figure out how to convince idiot industrial farmers that homeopathic water might be a better idea than pumping the livestock full of antibiotics. The problem is in filthy industrial farming operations, the livestock all die without antibiotics because ... drumroll ... filthy industrial farms are filthy.

Thanks for the insight. I've never thought of the possibility of treating the stressed-out parent with homeopathy. It seems brilliant!

My only problem is how does the doctor know that will calm the nerves? If my doctor prescribed my kid homeopathic medicine, I would never go to him again. Placebo only works when the people don't know they are being duped.

"Placebo only works when the people don't know they are being duped."

It's my understanding that this is (surprisingly!) not the case.

> Are there really? How common is "common"? This is the first I'm hearing of it, and I'd like more than the author's word. Perhaps a study? :)

Speaking from my own experience in Slovakia. It may be different in other countries.

> The answer has to be education - anything you can do, the "alternative science" crowd and profiteers can do, too. Unfortunately, neither I nor the essay's author seem to have a great cure-all answer for people who don't have the time, interest, or intellect to educate themselves.

I don't have an answer, but a suggestion: Let's make the processes such as publishing and licensing part of scientific method rather than a matter of political discourse. We can perform experiments to find out which processes do work and which don't. Once the successful processes become a part of the canon, violating them by a publisher or a licensing board can be pointed out as not compliant with scientific method.

This reminds me of another HN article this week, about trash journals and staged conferences. They accept any paper and give the participants a check-mark on their college graduation requirement.

In that case, its not hurting anyone (much). The experience is good for the participants, and if they're all sincere then some actual science may get done anyway.

"An obvious example: Sokal hoax. Physicist Alan Sokal have written an scientific article permeated with most obvious scientific nonsense and deliberate vagueness and managed to have it published in a scientific journal."

The journal was Social Text, a "cultural studies" journal. At the time, according to the same wikipedia page referenced in the article, Social Text did not practice peer review.

The fact that Sokal was able to get his paper published tells us nothing about the value of peer review in science

well, you can look up Schön scandal[1]. It would be a better example.

[1] https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal

I wasn't aware of that. Thanks for the link!

Simple rule, less about trust and more about science: the paper must be at least 5 (five) years old and have one (1) independent confirmation before I'll consider reading it.

It's possible to read the original papers and look for obvious signs of trouble, eg small sample sizes, improper randomization, etc. Basic science & statistics is not rocket science.

But the independent conformation rule is a really good filter. If, after 5 years, there's no follow up, there's good odds the paper was wrong. Or even if it's right, whatever it claims isn't considered important enough to follow up on. Even in the case of outright fraud like the Lancet autism article, the lack of independent studies reproducing the claimed correlation would have been a red flag.

Unfortunately, some very important papers wouldn't make it past this test...

The startup I'm working with has developed a unique, revolutionary product. (Unusually, this is a true statement - not marketing.) There are a number of published papers, and 8 year follow-ups. But journals won't even publish the 5 year follow-ups!

Why? Because it is a genuine paradigm-changer. We can't get independent confirmation, because no one wants to touch it. It threatens a global multi-billion dollar industry.

There's a 'mafia' in almost every profitable industry. Until I joined this startup, I had no notion that this niche was so full of medical practitioners readily sidestepping the Hippocratic Oath for the goal of profit.

That's fine, as far as John Q Layman's life is concerned, cutting through false positives with minimal effort. There will be false negatives with any rough filter. It sucks for you, but that's your fight to fight.

About the placebo effect: its not about what you know, its about what you feel. Enter a bakery, smell the cinnamon and yeast: your blood sugar responds. Doesn't matter if you know this is a placebo effect, it still happens.

So it's consistent that people in a non-blind test can still benefit from the placebo effect. Just going through the motions can affect your attitude, your activity level, your health.

For instance, I smile at myself in the mirror each morning. Changes my whole attitude. Doesn't matter how fake or forced; doing it and seeing it are a strong enough effect to mold my mental health. Even if I don't mind being a grouch myself, I do it because my family and friends benefit.

I'm a bit confused about your example with the bakery. That's clearly NOT caused by the placebo effect. Maybe you meant something else?

Ok, that was about how you feel about something, and how that makes your body respond. Placebos work that way (they have no other way they CAN work).

Or is everybody defining the placebo effect as just a bias in reporting (the good doctor gave me medicine; I'll say I'm feeling better in this questionnaire because I like that)? Because that's not the whole story.

People think you're confused because your examples were awful. The Placebo Effect depends on optimistic belief. Changes in blood sugar and attitude are involuntary reflexes. Involuntary reflexes occur regardless of how optimistic you are that your smile or the cinnamon buns will affect you. The exception where they intersect is Pavlovian Conditioning.

The “placebo effect” does not mean what you think it means. All those things you describe are real actual things and are not necessarily examples of the placebo effect.

Correct but you could have given him an analogy to help him understand.

A non-placebo analogy of "Just going through the motions" would be stretching and warmup before serious exercise, which provides a medically proven advantage to overall systemic performance, lower likelihood of injury, faster overall gains, etc.

Now a placebo of "Just going through the motions" would be skipping the exercise and watching an exercise DVD or perhaps watching a professional sports game on TV.

Yeah; it means when you put a non-therapeutic dose in place of a real active ingredient. But we were discussing how the placebo effect works. It works because you feel like it should work. Its not a conscious thing.

Some reading on placebo effect:

http://tedkaptchuk.com/selected-publications

My 2 cents on vaccines... Scientists are sometimes wrong. (Perhaps too frequently in medical science) Scientists sometimes fabricate evidence. (And when they do, they should go to jail, but that's another story) Usually the system self corrects over time.

Bottom line - if I have to decide on vaccinating my kids based on the advice of the general body of accepted scientific evidence, or a porn star, I'll go with the scientists.

There is always going to be signal loss from the raw data of the actual experiment -> the scientists quantizing and interpreting the data -> formatting and presenting salient parts of the data to get published -> popular science writers condensing the publication down to a few paragraphs -> general public interpreting pop science journalism -> word of mouth from your friends who read a pop science piece.

It's the age old problem of going back to the primary source. What exactly is the primary source? The raw data? The original publication? You can even go so far and claim the original experiment wasn't done correctly and you don't trust the experimenter. It's an infinite regression that just isn't very practical.

The key I think is more openness in scientific publishing and data. No more anonymous peer reviewers. Make negative data just as important as positive results...etc

On a free market, shamans and doctors invest their own time and money and can lose their savings if their method does not work. In the long run only precise scientific methods give reliable results while shamanism has random results. People vote with their money for the most efficient and better proved way to solve problems (provided kings, churches and governments do not interfere). Of course, this iterative process does not give you a short-term guarantee that you'll be healed, but neither does any sort of absolute authority.

We don't necessarily need a lot of people to understand a lot of complicated things. We just need to not interfere when some people think they can solve things better than before, even if you think they are mistaken. Doing otherwise creates a larger scale hazard (cf. today's patent and regulatory environment).

In the long run, we're all dead.

The market can remain irrational longer than you can remain solvent.

There is an unlimited supply of wrong theories; if yours isn't working, pick a new one.

"In the long run, we're all dead."

Speak for yourself, mortal. I'm taking this new immortality tonic...

A particularly dangerous approach to "Where was it published": The original Wakefield paper was published in The Lancet, which is as good as it gets. It was retracted by the journal, but that fact is occasionally omitted.

Unrelated to the article: I nearly jumped out of my seat when I saw a link to Martin Sustrik's blog on here. I recently discovered his blog and really enjoy his writing. He's one of the main programmers behind ZeroMQ and is now working on a new messaging library named nanomsg, using some of the lessons learned from building ZeroMQ. It's a great read if you enjoy the melding of technical and social writing. Pieter Hintjen's blog is also very interesting[0] for the same reasons.

0 - http://hintjens.com/

Thanks for kind words. I would like to point out the previous blog post (the one about immortality) that I am particularly proud of :)

There's an old quote from Larry Niven, I think, to the extent that, "If an old, white-haired scientist tells you something is possible, he's probably right; if he tells you something is not, he's probably wrong."

On the other hand, if a thousand old, white-haired scientists tell you something, it may be either right or wrong, but it's probably the way you should place your bets.

I don't understand that quote. Any scientist would tell you that perpetual motion machines of the first or second kind are impossible, and they're probably right. In general I don't know many examples of impossibility results in math or physics that were later overturned. Anyone?

It's just a maxim, it doesn't apply in every case. I think the motivating example is Einstein's fervent denial of entanglement in quantum physics. In other words, he's saying the tendency of scientists as they get older is to be more resistant to new ideas.

Whether that's true or not in general, is debatable. But it wasn't intended as an ironclad law in the first place, just a tendency

> Where to start: We are forever barred from knowing the composition of stars - stated around 1850, I believe. The sun couldn't possibly have sustained combustion for more than 10 million years, hence no time for evolution og geology. Parent poster is referencing Arthur C. Clarke, not Niven, btw.

ifix: You're probably right about Arthur C. Clarke. I think it was an SF author, but I couldn't remember who.

With some basic science training, you should be able to read the paper itself, see how they constructed the experiment, and have some idea if they did a good job. And this is without knowing much about the specific field. Did they have a big enough sample size, etc.

The press that reports on science could help with this too, but it is unfortunate they seem mostly useless in helping to provide a critical eye.

There are many more avenues for the non specialist to get a substantiated opinion. In this case, not just doctors, but the general press, scientific vulgarisation sites and blogs, facts surrounding the publication (all the authors but one rejected the study) etc. It's the combination of all these factors - these "evidences" - and not just one, that should convince people.

I found a bottle of Teething Tablets for my 10 month old labelled "homeopathic." I threw them in the trash, pointing out to my wife that placebo effect can only occur if the patient is old enough to comprehend that they are receiving a treatment.

The tablets were supposed to have placebo effect on your wife, not your kid :)

Didn't that make you feel a little better?

Why yes. Yes it did. Maybe the real benefit of homeopathy is making people like me feel superior... Hmm..

Heaven forbid we should start basing our scientific judgement on the severe numeric biases of Thompson Reuters.

When I point out the fact that science does not mean "just trust the experts, they are experts", and the scientific method does not mean "peer review by the anointed institutional authorities", the typical reaction is revealing.

The same people who bemoan a failing trust in our scientific institutions are themselves furthering it, by adopting starkly authoritarian anti-science attitudes. They compound the problem by rabidly attacking anyone for daring to question, which is behavior more fitting to a religious dogma than to a science.

"There are more things in heaven and earth, Horatio, Than are dreamt of in your philosophy." - Hamlet (1.5.167-8), Hamlet to Horatio

Reply on news.ycombinator.com