For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000?
Peer review is an absolute joke. In principle researchers could publish their data sets and reviewers could see if they could replicate the analysis but they don't do it because they know if they tried they'd get different answers, see
That was my thesis. To make it worse, the laser I used became unavailable as I was finishing up. My work was adopted by 2 labs but required extensive adaptation.
In the case of the LHC, which I assume is what you're talking about, this is why we have 4 pretty much independent detectors around the LHC looking at different but at least partially overlapping stuff. The Higgs boson was detected by the ATLAS detector, but wasn't considered to be "discovered" until it had also been detected at the CMS detector.
> Peer review is an absolute joke
That strongly depends on the field and the journal in question.
When I wrote my first paper somebody else had done a very similar experiment at another university and we wound method up peer reviewing each other’s papers. Turned out we were using the same bogus data analysis that was being used in 1000s of other papers. I was vaguely aware of this when I wrote the paper but did not feel psychologically safe at all in an environment where the American Physical Society claimed a PhD had a 3% chance of having a permanent career in the field.
Someone who was later to become a titan in the field was at our lab and was crying some nights because he had no idea where he was going to get his next postdoc wrote a paper about this problem more than a decade later after he had gotten tenure in a statistics journal that physicists probably won’t even read.
For a relatively recent example - in my field of quantum information someone found in the course of reviewing some newer work that there was a mistake in one of the big foundational papers that this newer work was based on. They shared it with some of their colleagues and together they did some work finding out exactly where the error was and what the impact is on work that is based on it.
The response from the community was they got a talk slot at a pretty prestigious conference (QIP 2023, you can watch the video here: https://www.youtube.com/watch?v=2Xyodvh6DSY) and the work they did has been published here https://quantum-journal.org/papers/q-2023-09-07-1103/ and in a followup paper here https://www.nature.com/articles/s41567-023-02289-9.
My point is essentially that it isn't a big shock that this error wasn't found by peer review when the foundational paper was published in 2008. Even when it is working as well as possible peer review can only detect most errors, not all errors. However when your field is functioning "normally" and everyone does their jobs properly this doesn't matter because the errors that get past peer review get detected by other people building on the work later. On the other hand if your field is not functioning properly and people are knowingly doing stuff that is wrong then peer review can't help you. If the status of the field is so broken that people who know stuff is wrong are still pushing for it to be published then you aren't in a place where peer review can save you.
My opinion is you are incorrectly putting the blame on the (highly imperfect) process of peer review here where it doesn't really belong. Peer review as we perform it today sucks for many reasons, but it isn't to blame for the problems you saw.
In medicine see
https://www.cochranelibrary.com/
where there is a systematic process to search the literature and select papers with usable results. I've never seen more than 50% of papers get accepted and the other day I saw one where they found 80 invalid papers and 2 valid papers. It seems to me that peer review isn't accomplishing very much if it is passing through papers that can't be built on. For that matter, why is this work even being funded?
This isn't an indictment of peer review as it is practised in every field.
Look at hep-th. If in 100 years people conclude there is no evidence for supersymmetry and extra dimensions something like 90% of that field will be “not even wrong”. There is so much of the drunk looking for his lost keys under a lamp. I mean, there is a huge industry in 2-d gravity where it is possible to calculate things but gravity doesn’t attract in 2-d so the line between that and Newton’s apple is unclear. I’ve talked to astro-ph refugees who think that current theories about accretion disks are “not even wrong”.
Then there are all the folks who tell me firewalls make no sense because there is no such thing as the event horizon (locally) but from the viewpoint of people who fall in it is a tiny difference getting killed at the apparent horizon meanwhile the nonsensical “black hole information paradox” (sorry, no unitarity = no quantum mechanics) still gets breathless and insufferable articles written about it just because somebody famous for their disability made a bet. If young people were able to get established in the field you wouldn’t see people so fixated on bad ideas.
If the results are interesting to anyone and there is more science to do, people will eventually replicate (or fail to do so, like with Majoranas). If not, then why would we care about replication?