> You say that "millions" of participants should be useful, almost automatically.
I don't disagree that there are issues with self-reported data, but I do disagree with saying that a large set of self-reported data is worthless.
I also didn't say millions, I said massive. Which in the case of research like this could be the difference of 40k participants (through an app) and 100 participants (clinically). While the size of the data set isn't everything, it does matter. With 40k participants you could aggressively invalidate 30k of them and still have 100 times as many data points for statistical validity as that clinical study. The clinical study would also cost more and take much more time to implement.
> How many self-report surveys would it take to match an actual double-blinded study with double digit participants? What if the answer is "hundreds"?
I don't know, but how many more can be easily done now if technology is used like it was here?
What if doing 100s in the same amount of time is now feasible?
What if it takes far fewer than your arbitrary assumption of 100s of studies?
What if it only takes 1 study with 100x the participants?
No one here is advocating for stopping double-blinded studies. Why not both?
> If you're off by an order of magnitude somewhere in your stack of assumptions, the data could be noisy enough that you need to be sample from a population of a hundred billion.
Why would you assume that it would be off by an order of magnitude? That seems about as likely as the data being perfectly accurate (that is, very unlikely).
And why 100 billion? It seems like you're just throwing out arbitrarily large numbers here.
> Should we wait a century for all these surveys to be done before even attempting data analysis?
This study took less than a year, and because it was automated, could likely be run concurrently with others. Why would it take a century?
> surveying the same set of people over and over won't give you additional data. You need to wait for some of the survey respondents to die.
Why would they survey the same set of people for the same study? There's a lot of people in the world.
I know you've decided for some reason that it would take 100 billion people, and that 100s of studies would need to be done to compare to even one clinical study. I know that for some reason you think that would take a century instead of in parallel.
Consider for a second that if you're wrong on any of those magically large and arbitrary numbers, this might actually have some value to the world instead of dismissing it out of hand.
Again, no one is suggesting any fewer clinical, double blinded, etc. studies. Doing studies this way has some potentially very large upside, and it can be easily used in tandem with those other studies to speed up research and gain much larger data sets.
Yes it has trade-offs. We can be aware of those without throwing out the whole idea.