In my opinion this is an exemplary N=1 study that is well designed and thoughtfully executed. Deserve accolades, not derision. And the author even recognizes possible improvements.
Unlike most large high N clinical trials this is a high resolution longitudinal trial, and it is perfectly “controlled” for genetic difference (none), well controlled for environment, and there is only one evaluator.
Compare this to the messy and mostly useless massive studies of human depression reviewed by Jonathan Flint.
Anecdotal data, n=1 trials of varying quality, correlations studies, double blind studies (with small and large cohorts), studies without attempted replication and studies with heavy replication - they are all provide evidence of varying quality and can inform the holistic scientific picture. They can all serve a purpose such as inspiring further research, providing fodder for meta analyses, etc. It simply isn’t true that gathered evidence ought to be casually discarded if it doesn’t attain the highest levels of the hierarchy of evidence. Neither is it true that some small study showing (or not showing) some supposed effect should drastically change all our lifestyle habits. The truth lies somewhere in the middle. The concept of a hierarchy of evidence can help us navigate these apparently mixed signals so prevalent in popular science discussions.
We should avoid extreme polarization of our judgments in general. The study deserves some amount of praise for things it did somewhat well (like the method of blinding which is clever, but not applicable to everyone), and criticism for things it did not do well, such as designing your own study methodology for your own mood. That alone will affect the results. Simply RUNNING an experiment can affect your mood because it's interesting (or even maybe frustrating). The subject probably felt pride and satisfaction whenever they used their pill selection technique, which could improve mood on its own. Neither accolades nor complete derision are appropriate, although trying to claim too strong a result from this study is kinda deserving of derision if you claim to be science-minded.
The study was well-meaning and displayed cleverness.
Also, you asked why he said "unblinded", and I think you now have the answer to that.
Re-reading such a negative critique of a solid home-brew experiment is unwarranted. There are several word here worth red flags.
>This is an N=1 trial. Dressing your N=1 trial up with lots of pseudo controls and pseudo blinding and data collection does not make it better. In fact: putting this much effort into any medication trial makes it much more likely that you’re going to be incentivized to find effects that don’t exist. I think it’s nice that the author admits that they found nothing, but statistically, worthless drugs show effects in much better-designed trials than this one: it’s basically a coin toss.
>While I was blinded during each trial, I saw the theanine/D result when I wrote it down. Over time I couldn’t help but notice that my stress dropped even when I took vitamin D, and that I was terrible at predicting what I’d taken
That is not blinding
Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless.
As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment?
Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.
It gives a hint of how well the blinding worked.
> no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location
Assuming the blinded samples are uniformly randomly distributed, and assuming the study goes on long enough, then you'd expect that stuff to average out.
But I agree, it should be recorded nonetheless. That way you can verify at the end that it did, in fact, average out as you expected. If it didn't then your data is invalid.
That would not be a problem regarding the averaging I referred to, although it could well pose a problem for measurement depending on how it interacted with the selected metrics.
Note that the averaging I refer to is not regarding all possible values of some metric, but rather any discrepancy in the distribution of metrics which we expected to follow the same distribution between the sample and the control.
I think maybe there's a misunderstanding? It seems that we both agree that a variety of additional variable should be logged. I was not suggesting to omit them, but rather to use discrepancies in them to detect fundamental issues with the data or study design. I would also expect larger studies to do the same where possible.
At 94 data points it is entirely possible that there would be outliers that would have averaged out for a larger N but did not. In such a scenario the presence of such outliers should then be taken to indicate a problem with the data (ie the more discrepancies you observe, the less you should trust the data).
I of course agree that logging them is basic scientific methodology - in order to detect issues with the experiment, and even hopefully to see the signal through the noise.
It’s unblinded because the subject is preparing the concoction under study. There is no way they can create a blind experiment if they’re the ones preparing the control. The placebo effect is nothing if not pernicious and cunning, able to exploit even the most subtle psychological signal - like minuscule differences in the amount of powder in a capsule.
Blinded studies have independent doctors prepare and dispense the candidate drug so they know whether its the real thing or a placebo, but their patients dont. In double blinded studies, neither the doctor nor the patient have any idea about what they’re getting because a third party prepares the drugs.
I think most people could criticize the carbon out of a corpse if they themselves weren't being criticized into one.
If we devolved from apes, maybe apes devolved from piranhas.
It is clearly a step forward from what you can watch about theanine on YouTube or TikTok. I consider this a work of citizen science. While it should not be taken for more than it is, it’s a great example of how someone can experiment without a high burden.
Comments like yours expose a particularly distasteful amount of hubris.
The New Yorker's paywall has successfully obscured the origin of the joke, so that's on them, as far as I'm concerned.
Well, instead of an actual web link, you could just mention that it's from the New Yorker.
(From a vaguely-remembered 1990s-era ad campaign that I thought was excruciatingly self-indulgent at the time, but which evidently worked.)
This is only an appropriate criticism in so far as you want to make conclusions about theanine as an intervention in the broader population.
It is however, perhaps much BETTER than large N trials if the author wishes to draw conclusions about how theanine affects THEM.
Now you can argue that there isn’t enough time between samples, or he needs more subjects but he was blind to whether he was taking it that day or not.
Sure, OP addressed that and said it would be especially useful more people did it and "if we find a way to collectively interpret the data".
How else are you going to find out whether a particular diet or medication works for you specifically? It's ALWAYS N=1.