Hacker News Reader

Comment by nonbel | Hacker News Reader

>"The only way to measure the size of genuine placebo effects is to compare in an RCT the effect of a dummy treatment with the effect of no treatment at all."

This is just wrong, the following procedure would be much more convincing than a RCT:

Decide what you are measuring, collect group of people A, measure it, give people placebo, measure it again, record results. Then repeat under different circumstances with a different group of people B. Maybe even go back and do A again. Are you getting consistent measurements? Good.

Now come up with an explanation for why the results have that distribution, sources of variation, etc. Use that to quantitatively predict what should be seen a new group of people C. Now go check group C. Did it match the predictions? If so, good, keep at it.

Make a prediction for group D. Do the group D results match? If so, you are probably interpreting the results right.

View on HN

How does that address the get-better-anyway effect, without a group to measure it?

Because you are predicting precisely how much better can be accounted for by the placebo effect, no more, no less (of course within error bounds).

The get-better-anyway effect is too vague. Say theory A predicts value x is in the range [3.1,3.3] and you observe x=3.36 +/-.1. Also, there is a theory B that predicts x>0. Theory A has been much more severely tested so is supported by the evidence. Theory B, meh.

Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result. Then there's also theory B (the get-better-anyway theory, or null hypothesis), which wasn't measured initially but in recent studies shows a similar 25% improvement.

I don't understand how more studies testing theory A and getting 25%, while excluding theory B from testing, can show that A should be preferred over B. The rationale that A has been tested more thoroughly and therefore has a smaller confidence interval sounds circular.

>"Say theory A (the placebo effect theory) predicts a 25% improvement, because it's been measured in the past with that result."

That is not a theory. You have to come up with an explanation (a theory) for why it should be that value. Then from that theory you deduce predictions for what it should be under other circumstances. If the predictions are consistently close to the observations it indicates you are onto something.

The discussion is about how to test the placebo effect (which corresponds to your example's "theory A"), so the proposed alternative to RCT would have to include the process of first deducing the effect size. For this problem, how would you realistically do that without referring to past experimental results? There are so many unknowns, including psychology. The 1955 paper was itself a survey of experimental results.

I expect it to be much easier to come up with a statistical prediction for the get-better-anyway effect (regression to the mean), with more specificity than x>0. It would be needed regardless, in order to exclude the null hypothesis.

My proposal includes collecting experimental data. It is just not in RCT form (which was claimed to be "the only way").

There is no reason to ignore RCT results, but that is not the only way to get the necessary data (which is what I disagreed with). An RCT is also not sufficient on its own, you need the theory that explains the results as well.

As an untrained outside observer (but one who has been thinking a lot recently about the statistics of causality) I'll chime in to say that I'm siding with 'ashearer' here. As I'm reading it, your approach doesn't make sense to me. If you want to distinguish between two treatments (placebo=treated and do-nothing=untreated) you need to measure outcomes for both treated and untreated individuals, and these individuals need to be "exchangeable" to some degree.

Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"? I agree that you can often draw useful conclusions from an observational study as long as some comparable individuals receive each treatment, and as long as you can control for the initial differences between the treated and untreated. But if you are saying that you don't need to look at the results for the untreated group at all, then I think you are mistaken.

>"Perhaps when you say in the first post "give people placebo", you mean "give the placebo to some subset of the individuals leaving the remainder untreated"?"

I meant just placebo. I am using predictive skill of the theory behind the explanations to distinguish between them, this is common in physics/astronomy/etc. As noted by asherer, placebo effect has been studied for a long time in the manner of seeing how two groups differ. There is nothing wrong with RCTs, but does near sole reliance on them (and their observational analogue) appear to have lead to cumulative knowledge and growth of understanding?

Both hypotheses in this case are plausible. Beyond that, it would be great to have the tools to numerically calculate a psychological phenomenon such as the placebo effect the way a physicist calculates the energy of a particle. But we're very far from having them, and in the meantime, I don't know of a practical alternative to RCTs that would expand our knowledge at least as much. (I think we can both agree that over-reliance on significance testing is a problem, though not one limited to RCTs.)

Check out the paper I posted here earlier: https://news.ycombinator.com/item?id=10556521

Reply on news.ycombinator.com