On Reddit's /r/science board, there's a base rule that "criticism of published work should assume basic competence of the researchers and reviewers".
I think it would benefit HN if we all made the same assumption.
That said, if you are going to criticize something based on the idea "this result doesn't agree with my preconcieved notions so I'm going to say that their sample size was too small, the effect size was too small, or they made some sort of incorrect calculation, and conclude the paper can't be trusted", you owe it to yourself to read the paper carefully enough that you can make the determination they made a major error.
Unlike most people, I read the conclusion, then the methods section. For most papers I can't get enough from the methods section to conclude that I trust the authors to make their claim, because most methods sections are missing major details on how the corrections/controls were made.
I believe the latter, and the rule I mentioned also leads to that assumption. Otherwise, why read the studies at all? But that's not what people here do.
> missing major details on how the corrections/controls were made
I also make an assumption that some corrections/controls are so obvious within their field of study that they aren't worth documenting - that it's only "average joe" readers who assume they aren't being done.
It is, in a way, akin to asking why programmers didn't document how their JWT is secured.
BTW I've definitely gone into code reviews assuming the programmer lacks basic competence and been right. Other times, I've had to rollback other people's code (or stop a rollout) because an ostensibly genius programmer who made a change (and got a review from a starry-eyed junior) wasn't using their basic competence or testing their change at all.
The former is perfectly fine and encouraged. The latter is incredibly rude and derails discussions.
Relatedly, you took the rule incredibly literally just so you could critize a strawman...
> Unlike most people, I...
Yes, yes, you a very smart and everyone else, in particular me, is an idiot...
Put another way, just because Cosmo publishes an article doesn't mean it's true - but Cosmo does also publish true articles.
Plenty of studies are great and move our understanding forward. More studies answer a very narrow - but important - question. Some studies are just inconclusive, but have value too.
You should read studies. You should evaluate the quality of studies as part of integrating them and their conclusions. At the same time just because something was "a study" or got published doesn't make it true or good or useful, anymore than being written in blue ink makes something truthful.
The value of science is that it does not require faith.
If someone were advocating making policy decisions based on astrological charts, I would read and comment in such threads despite not trusting astrology on a fundamental level.
I personally would go quite as far as to say that purely observational studies have as much predictive value as astrology, but "observational studies are as reliable as astrology" is a better first-order approximation of my opinions than "observational studies are as reliable as large-n randomized controlled trials".
With N different control variables to either include or ignore, that's 2^N possible sets of control variables. Odds are decent at least one of those regressions has a large effect size for coffee.
I would trust this sort of research more if instead of publishing a particular set of control variables obtained by an unspecified method, the researchers chose 100 of those 2^N possible sets of control variables at random, then published the average effect size from the 100 resulting regressions. Ideally they would make the code to reproduce this average effect size publicly available, so anyone could easily replicate using another 100 randomly generated regressions.
Since most studies/publications cant be replicated, I'd say basic competence is not warranted.
Honestly - I do. Basic competence in creating a study would definitely include identifying confounding factors that are directly relevant to the study being conducted. It's a basic part of the scientific method - both in creating the hypothesis and creating a test against that hypothesis.