> You can't run a randomized control trial in this fashion, which makes it harder to collect meaningful data.
You can, it just needs to be organized properly. You sign up for a test of a new drug/nootropic online, you get the pills, but you don't know if they work or not.
Then, you have to report the results.
> * You can't select the populations of people who are in your test groups. It's possible the drug will only work well in certain populations.
First of all, you can. Require filling a questionaire (or even genotyping) before sending the drug. Not only you will be able to analyse the statistics across populations, but you will no longer have problems with many drugs being only tested on white males.
If certain populations are missing from the data, the company could afford the therapy to them for free, or even pay them.
> * You can't monitor people as well for things like adherence or control their other behaviors. Perhaps they are taking 5 other compounds simultaneously which are having an effect or interfering with your compound of interest.
What has a higher chance - experiment subject lying about other meds he's taking, because he wants to get the drug (or money for participation in experiment), or a person who stands nothing to gain or lose from filling in the questionnaire truthfully?
> * You can't screen for toxicity or determine dosing meaning the drug could kill people en masse before the ideal dose is determined.
The idea would be to allow only drugs that passed initial trials to estimate toxicity and safety. But open the drug to public before efficacy is established.
By the way, every single day doctors prescribe drugs with "unproven" efficacy. While every drug had to be proven that it's good for "something", many of them have off-label use, which is currently totally not tracked.
I was prescribed Modafinil for ADHD. It's off-label, and there is some research proving that it helps. I feel it's helping me too.
Unfortunately, I have no way of filling any kind of detailed description of perceived effects, or my history. Nobody offered me an option for participation in double-blind study. There is so much lost potential there.