And it then turns out that most methodology even in reputable journals are rather wanting with many objections that can be leveled against it.
A large amount of scientific research can't even be reproduced, and of much that can, even though the cold data can be reproduced, the conclusions that follow from the data are rather dubious leaps of faith.
It doesn't take much for something to be called “science”; it certainly doesn't take reproducibility, despite various claims to the contrary.
There is a lot of noise already in "proper" journals - but in the predatory journals, the signal-to-noise ratio is so extremely low, it's not worth looking into the credibility of the methodology of the paper because that's far more time and effort that the paper deserves. If it was any good, it would have been published in a better venue. If it could pass peer review, it would have been published in a venue that actually does peer review as opposed to these (many) predatory journals who just claim to do so. The authors have strong practical incentives to not publish it there if they can avoid to, and the fact that they chose to do so anyway indicates that no respectable place would publish it.
Because of that, if a paper is published in a place like this, is a completely reasonable prior to presume that overwhelmingly likely the paper is very bad, without even looking at the paper itself.
But how the post which I replied to was worded suggested that one can automatically trust “science”, provided that it be published in a credible journal.
I also find that very often in the most reputable journals is where the most sensational papers end up before any attempt at reproduction has been made simply because the data they measured was far from the null hypothesis, which may entirely be a statistical fluke and not hold up under a reproduction attempt.
It is really quite easy to obtain spectacular data as a fluke.
But you still have to look at the individual paper and can't believe it just based on a single article. Research articles are for sharing results among experts (and for advancing the careers of researchers), they are not aimed at laypeople. It's easy to misinterpret them if you lack the background knowledge.
Even in non-predatory journals many results fail to replicate and are produced due to a publish or perish pressure. You're better off learning from textbooks so you get info that has been verified, digested and distilled and represent consensus. Cutting edge research proposes new ideas by one group of authors, it's not a consensus yet.
A very large body of consensus ended up in textbooks that had either never been attempted te be reproduced, or was attempted, much later, and couldn't be reproduced, yet continued to remain in textbooks.
The truth of the matter is that most scientific research will never see an attempt at replication, though the replication crisis has no doubt influenced this culture, but before it, as little as 0.16% of peer reviewed results were attempts at replication, and most were unsuccessful. — of course, the successful ones were not as easily published, which is probably why no one attempted it.
Regarding replication... Replication is boring in the eyes of funding committees. They want new and sexy results for their money, preferably at a steady rate of a bunch of papers each year.
Beyond that you can look a bit at the impact factor of the journal, you can generally find it by just googling for the name of the journal + "impact factor" or on the Wikipedia page of the journal. But it is kinda hard to interpret this on its own, and it does not translate directly into credibility. A high impact factor shows that articles in that journal are cited very often, and you can assume that journals with high impact factors generally have a reasonable peer review system. But that doesn't say much for a single paper, reputable journals can easily fail in individual cases.
The Pubmed check generally filters out most of the journals that would just publish anything without real review, so it's the most useful filter in my opinion. I would not try to gauge more meaning than that from the journal and focus on the individual article.
You could probably gain an entry level background (except for highly mathematical or medical things) by spending 5-10 hours a week for a few months reading various papers/online discussions. Assuming you have access to those forums. You should know thats the lower bound amount of literature reading full-time students have to do to keep up with their own field.
Nobody keeps reading full time, that would not only be insane it would also be conterproductive since it’s time not spend actually doing something or writing papers. In most place it’s not necessary to know all the recent papers, and if a really important one is missing as a reference, a reviewer will note it in the comments.
Articles are peer-reviewed for their readability (for peers, not laymen), consistency (e.g. if there is a mathematical proof in there, the reviewer might check it if possible. if there is an experimental technique, the reviewer might check if that technique could produce the results claimed) and reproducability (is the experiment described in sufficient detail? does the software provided produce the expected results? do the provided numerical results fit the claims?). All those checks are what the reviewer can do in a few hours, days at most. That is what peer-review means.
All the things to check beyond those are left as an exercise to other scientists after publication. Reproduction is a big task, usually almost as big as the original paper. Proofs are often extremely hard to check, so a few days by any old reviewer won't cut it. So if you want to know if the results in a paper are true, watch literature for the following years and look at publications citing that paper and (dis-)agreeing with its results. Same for (sometimes) letters to the editor, retractions and "everybody-knows"-rumours.
More reputable journals tend to attract better-quality papers (according to the peer-review criteria outlined above). That tends to correlate with a higher probability of the paper being true. But it is not a very strong correlation, there is utter nonsense, groupthink and polished turds even in very high profile journals.
I mean it's not totally arbitrary, really good reviewers do give it like 2-5 hours, but it's best thought of as a rudimentary filter rather than a meticulous verification.
I don't think blindly following these metrics is a good idea, but it's not as a bad first approximation.
[1] https://en.wikipedia.org/wiki/Journal_ranking
Most if not all scammy journals will fail this test.. but it is also quite labor intensive.
It's really tricky, because the move to open access publishing (which typically requires that authors pay) is a good thing on one hand, but facilitates these junk journals (and inflates publication count) on the other hand.
There are also Cabell's lists, both of predatory and "quality" journals.
Nature publishes many journals and "Nature" is the highest quality general science publication. "Scientific Reports" does just quick review for methodology.
It isn't perfect. Different disciplines tend to cite at different rates; and you may believe that some disciplines are just cabals of bullshitters citing each other. I couldn't possibly comment.
Then you can judge the arguments of the paper and the papers referencing it.
Usually a paper with thousands of citations and only weak arguments against it is trustworthy.