Hacker News Reader

> > it makes sense to look at change from baseline for individuals in the various groups rather than just differences in raw scores between treatment groups.

> That’s called cherry picking. Results are always noisy. You could give two groups of people the same tests on different days without any drugs at all and some subset would show “improvement”.

No, GP is talking about a matched pairs design. You look at the difference between the scores of very similar individuals by applying one treatment to each (active and placebo), or, applying both treatments to the same individual (in random order).

Cherry-picking would mean only using scores from selected individuals, whereas matching only emphasizes the difference.

They're not saying you should look at some subset of individuals with positive results, you have misread.

The notion is that the difference between participants can mask the effect of the drug, such that comparing any individual participant to anyone but themselves is improper.

No, why would you claim that?

Claim what? If you go through the results and exclude the individuals who didn’t show the outcome you wanted to see, that’s called cherry-picking. It’s a well-known phrase.

There is a concept of subgroup analysis in studies like these, but you have to be careful about how it’s done and what conclusions are drawn. If you simply select positive results and exclude negative results then even the placebo group would show great success.

This study showed that telling people that they were microdosing was more important for the perceived outcome than the micro dosing itself. In other words, placebo is key to making it work.

> If you simply select positive results and exclude negative results then even the placebo group would show great success.

Where did zosima ask to do that? They mentioned that the variable of comparison should be changes from baseline metrics in treatment group vs changes from baseline metrics in control group. That would be a fair study, and isn't cherry picking.