It being extremely difficult or impossible to design a convincing study doesn't make the other ones convincing.
Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist.
So, if posed with that question, I would not have a good response. Despite this, I would still believe the dropped piece of paper experiment was not very good.
I am allowed to ask, right? It doesn't seem crazy to me that when someone criticizes a study, someone else will ask "how could you solve the problem you describe?". I would like to know if someone has any idea on how to falsify it, because if someone were to find one, it might convince me of the criticism.
> Let’s apply a similar logic to something that is clearly possible to design an experiment around. How about detecting neutrinos? If we were commenting on an article about detecting neutrinos by dropping a piece of paper and seeing if it bounced around a bit on the way down, I think we’d all identify that as not a very good experiment. But, I could not design a very good experiment for detecting neutrinos. And, this doesn’t say much about whether or not neutrinos exist, I’m just not a physicist.
I don't think that is a good comparison. A better one would be if there is an experiment, and then someone comments on the results by saying "well, it could have been dark matter particles interacting in some way". It might as well be true, but it's not definitive if they don't explain how to measure and isolate that. Someone else asking for how they would design the experiment to take this into account seems like a completely reasonable ask to me.
They'd need to be in a situation where they believe that the statement of their true beliefs is critical.
This isn't impossible (many studies do use primary sources which were not created for the purposes of a study, and where there was some incentive based on real-world circumstances for them to be truthful), but it is much more difficult and leaves gaps where insufficient primary sources materials addressing the questions of the study are available.
Psychologists are well aware that the current model for many psychology studies, which consists of either asking an extremely small sample of participants to sit in a computer lab at a university for 20 minutes answering questions, or sending out very loosely-targeted emails soliciting participation, are both very inefficient.
But they gotta get published, cause that's how you put food on the table.