Hacker News Reader

Comment by knzhou | Hacker News Reader

I don't want to sound too triumphant, but over here in fundamental physics we have decades-old safeguards in place against all these problems.

The big experiments don't have publication bias: they proudly say exactly what they did, even if 90% of the time there are only negative results, because exclusions are important too. Experiments are inherently replicated, with multiple independent simultaneous experiments (LHC) or multiple independent analyses (EHT), data blinding throughout, and even occasionally a further layer of blinding using decoy signals (LIGO). The statistical standards for discovery are, in terms of p-values, about 10,000 times more stringent, and even still people are moving away from p-values entirely.

The resulting publications are put out for free, publicly, on the ArXiv. Later they are submitted to a relatively small family of low-cost journals, which everybody knows the reputations of.

Hopefully some of these lessons can be adapted to other fields.

View on HN

>The statistical standards for discovery are, in terms of p-values, about 10,000 times more stringent

But this has mostly to do with your subject matter being better behaved than others, no? The most profound changes and the healthiest research culture could not make 5 sigma a reasonable goal in psychology.

Our statistical standards may be unrealistic for social science, but there was a petition a while ago to lower the cutoff from 0.05 to 0.001. I think that really would help.

There's also a very healthy question of weather or not it's worth while. A p-value that's theoretical physics stringent will mean leaving a lot of cures on the shelf.

While it's certainly more robust, it still has some issues. For example, I remember reading that because the initial results for the experiment measuring the size of the electron was too low, every experiment afterwards reported slightly bigger sizes until they reached the correct value.

Since they trusted the original result, if their results varied too much they discarded them. Ultimately they redid the experiment until they got something closer, and the only values that ended up getting published were results that were slightly bigger.

Correct me if I am wrong, as I read this story a while ago.

That's an old Feynman story, but he was kind of bullshitting to make a point. If you actually look at the history of measurements, it doesn't look like that at all -- it instantly corrects to about the correct value right after Milliken and hovers around that value.

https://hsm.stackexchange.com/questions/264/timeline-of-meas...

Actually if you look at the follow up answer, they added some more measurements that back up Faynman's claim.

Huh, I didn’t even know! The new answer really does change things.

There is similar example for the speed of light as well.

This is exactly the sort of thing that blinding is there to prevent. It is the same reason we in cosmology blind our data - to prevent us from biasing ourselves towards what other experiments have found.

Also, the charge of the electron experiments you are talking about happening over 100 years ago, so not that relevant to today :)

I think you're thinking of the oil-drop experiment for the charge of the electron that was done 110 years ago (1909). And the experiments in the subsequent decades. Feynman, in 1974, so 45 years ago, used it as an example of problems in science. So, in response to the original post saying that particle physics has been doing things more or less correctly for decades is not tarnished by your example.

More details here: https://en.wikipedia.org/wiki/Oil_drop_experiment#Millikan's...

This is where measurements by multiple groups come in. I would point out that in 2019 the basis for the SI base units were changed because of issues with the standards. The international community - lead primarialy by NIST and their counterpart agencies in other countries noted that, among other standards, that measurements of the "standard meter" bars were changing measureably and the replicates of the "official bars" differed from one another.

They proposed and agreed upon new "standards" based upon measurments with better precision and accuracy. See the Wikipedia summary [1]

[1] https://en.wikipedia.org/wiki/2019_redefinition_of_the_SI_ba...

Seems like a bias-variance tradeoff. It's not obvious to me that what they did was bad.

Yeah but then you have different problems.

Hundreds of authors on each publication. Whose contributions are real and who is just cruising?

Maybe this just shows that using authorship credit as a proxy for contribution is a bad system.

I mean, can't you say that for any large group of people? You can't easily evaluate who's pulling the most weight at Google either. But that's not a problem, because the people at Google deciding promotions do know, and that's what matters. CERN works like that, as does experimental physics at large.

Most positions at CERN are fixed time. Afterwards you have a problem.

If you leave CERN to apply for a postdoc or professorship somewhere else, they ask for lots of recommendation letters. That's how they learn what you actually did.

Do you think a recommendation letter is a more objective and better evaluation of what someone actually did? That there is less embellishment, less subjectivity in it than a scientific paper that was meant to reflect on your work as a scientist.

I would expect that recommendation letters heavily favor some individuals for subjective reasons.

As does hiring when a Googler goes to some other company. Nobody is going to see their commits. They have to go on their word, their technical skills, and their references.

The fact that there are other instances when people make subjective decisions is not quite relevant in this instance.

We are talking about the case when authorship on a scientific publication is insufficient evidence that someone had any contribution whatsoever to the paper. So that one needs a recommendation letter, in addition to the authorsip, where the recommendation letter would presumably state, this individual did actually do some work and is not just cruising.

No, in Spain you do not have that choice...

At the LHC the number is closer to 3500. I "cruise" on around a hundred papers each year: I haven't read them and in all likelihood never will. At the same time I worked really hard on one or two of them.

I have tried to figure out what would be required to remove myself from 98% of the papers we publish, but it turns out to be a lot of work (essentially you have to argue with the head of the experiment, who has other things to worry about).

this would be tough for me; I will not allow my name on a paper unless I read it before publication (and had some sort of useful, direct contribution). I guess large physics collaborations are very different from small bio projects, but still, I wouldn't want my name associated with a paper that later got retracted for something I would have caught.

Well when you do your PhD in particle physics it's just sort of accepted.

And I think it's tough on a lot of people: many people in the experiment feel that they should read every paper, because they feel personally accountable to it. They feel they should insist on changes because otherwise it will reflect badly on them.

To me that's an enormous amount of work and gets in the way of the really interesting science. Within the field everyone knows that I'm not directly accountable for a paper just because my name is on it. Most work is done by small teams of less than 20 people (sometimes just one or two) so it would be absurd to ask me to fully understand, much less feel accountable for, 95% of what comes out.

Personally I'd love an opt-out option. I don't think my colleagues should feel burdened by any weight that my name adds. Beyond that it's a convenience thing: I was updating my CV and thinking "if only there was some automated system to keep track of the papers I contributed to...". Incidentally, we have such a system, but it's only visible within the collaboration.

Determining attribution for scientific contributions is a far less severe problem than having high-quality scientific contributions in the first place.

> 90% of the time there are only negative results

That’s an understatement.

What people forget is that this was true in the old days as well. Even back in the 60s and 70s there were over ten accelerators throughout the world constantly doing science, and you can bet they were booked up with new speculative searches every month. The vast majority found nothing, and were forgotten. That's because there's a very finite number of things to discover in fundamental physics, which is of course what makes the field so exciting in the first place.

Of course. It’s actually pretty awesome that every time the energy frontier was pushed, something new was discovered, so there was return on every round of investment.

cought 750 GeV cough... ahem. Let me sing you an OPERA about a how Andrei Linde was astonished by the biggest BICEP you've ever seen! It belonged to a LIBRA* that was faster than light and could fly trough walls!

* Couldn't get DAMA to work :(

Sure, but there's a difference between the accuracy of speculative theory and experiment. Experiment is very reliable -- they never called the 750 GeV bump a discovery of anything, because it didn't meet the statistical standards. Even things that do meet the statistical standards are often not called discoveries, because the experimentalists think they are systematic errors at play, as we see with MiniBooNE and DAMA.

Speculative theory is not reliable, which is why all the papers on the 750 GeV bump were wrong. But it has never been reliable, because, well, it's speculative! For each new phenomenon there's probably only one right explanation, but far more than one paper. Being 99% wrong has been par for the course for almost a century.

I mean, if CMS just hadn't played up their 2.3 sigma excess we probably wouldn't have had the 750 bonanza* so don't pretend there is no blood on the experimental hands :p

OPERA is a bit of a special case, but BICEP just straight up declared discovery and wanted the Nobel, have you seen their video of when they tell Linde?

* Any time I can, I link to the conclusion: http://resonaances.blogspot.com/2016/06/game-of-thrones-750-...

> BICEP just straight up declared discovery and wanted the Nobel, have you seen their video of when they tell Linde?

That’s a bit exaggerated. You mean the champagne popping video? Well, it’s somewhat awkward in hind sight, but a bit of excitement was warranted at that time. I was at Stanford around that time, might even talked to Andrei right after, and don’t recall a festive atmosphere or anything.

I feel bad for his wife every time I see it now :(

She’s also a Stanford physicist. Great person.

I agree that BICEP was the worst debacle of this kind that’s occurred in recent memory. But also see how the safeguards worked: their results were properly questioned, the hype was over in months, and they did not get a Nobel.

The high profile debacles were pretty quickly caught, so I’d say the system is working.

There are two aspects: how many errors are published and how they long before they are caught? The BICEP debacle is an example of a claim that should have gotten caught before it was published, doing an important calibration step using a photo of preliminary data?!

Actually, the error was caught before publication. They sent it to ArXiv, the standard pre-publication space, so that other physicists could read and comment on it. It was there that the error was quickly caught.

Oh, I didn't know that. I though with all the announcements they did they had a paper ready to publish...

Well, it's all a little ambiguous. You can choose to pre-publish or not, and you can choose to announce whenever you want: before pre-publishing, after but before publishing, and at the moment of publication itself.

It's certainly true that they announced early, but it's also true that the community at large regarded it with appropriate skepticism, causing the whole thing to be self-corrected in months.

"Stanford Professor Andrei Linde celebrates physics breakthrough"

https://youtube.com/watch?v=ZlfIVEy_YOA

>> Hopefully some of these lessons can be adapted to other fields

the primary lesson here is to get a commitment for decades of government funding

Preferably in the billions of dollars. Nothing generates five sigma confidence levels like ten-figure budgets.

Sorry, do you think that fundamental physics gets more than other fields? Have you ever seen a breakdown of research budgets?

I don't have such a breakdown, but I do know that the budget of the LHC alone is a billion dollars a year plus $13B to build it. That wouldn't have been possible if it weren't for a long-term commitment.

And I know that few fields get that kind of commitment. Maybe cancer, maybe something in semiconductors. Lots of other fields would produce things if you offered them many years of billion-dollar commitments.

I'm not saying that it was wrong to give so much to fundamental physics. I'd just be very happy if we gave that level of funding to lots of other things.

CERN and the LHC are to me prime examples of bad science. You build huge machines at incredible cost, wasting hundreds of thousands of man hours of brilliant young people’s time. They are paid almost nothing (a checkout clerk at a Swiss supermarket makes more than a PhD student at CERN). Then you string along a subset of those for years, exploiting them for further cheap labor, somehow making them believe they are “lucky” to get that opportunity (a postdoc at CERN pays a fraction of what you can make at Google Zurich). In the time between experiments people only ever see simulated data, leading to a rude awakening when actual experimental data comes in (c.f. ALICE’s desaster of a analysis Pipeline). Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter). Defunding the super collider in the 90s in the US was probably one of the best science policy decisions they made.

I kinda wish this comment focused more on bad science rather than bad pay. If bad pay implied bad science then most of science is bad!

I suppose I'd be much more interested in learning about the latter half of your comment (specific scientific failures).

> Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter).

Every statement in your diatribe would seem to be predicated by the assumption that discoveries in theoretical physics (what you're complaining about) will result in revolutionary changes to your day-to-day experience (kinda i.e. applied physics) in a short enough span of time that you can actually enjoy them with an able body.

Probably not. Flight/space and semiconductors were notable exceptions, and they were largely driven by the century of hot and cold war that we know to be the 1900s.

Bad science is science that fundamentally doesn't yield a new understanding of the world, either by falsifying theories or confirming them. What you described is:

1. failure to meet exaggerated lay expectations, and

2. poor working conditions in the nonprofit space.

#2 (and possibly #1 depending on where the theoretical discoveries take us) can largely be resolved by pouring more money into science, not less. I'll point back to our century of war if you need past precedent.

Edit: removed a line that was unnecessarily insulting. Sorry about that.

Indeed, flight and space have applications, which is why they (completely justifiably) get more funding. People seem to forget this, but the space shuttle alone was funded over 15x more than the LHC was, and by a single country rather than many.

> Every statement in your diatribe would seem to be predicate...

I would be happy if they found any experimental evidence for their predictions. If you have followed the field basically the majority opinion was that supersymmetry at the LHC would be inevitably found, as long as the Higgs mass was in a certain range (naturalness argument). Turns out it (narrowly) was in that range but supersymmetry was not found anyways. Arguably a lot of particle phenomenology in the last 20+ years was bad science in that way, because no new phenomena had actually been observed/measured and needed model building. Of course there is nothing inherently bad with conjecture or theory, but given the fact that there are lots of unsolved problems in physics on which progress was made in that time, a lot of what they were doing seems like a huge waste of time.

Philip Anderson made his case for refocussing physics a long time ago: See his case against against the SSC: https://www.the-scientist.com/opinion-old/the-case-against-t... and "More is Different" https://science.sciencemag.org/content/177/4047/393.

> Arguably a lot of particle phenomenology in the last 20+ years was bad science.

Perhaps that wouldn't have gone on so long if the SSC wasn't fucking cancelled in '93.

> Arguably a lot of particle phenomenology in the last 20+ years was bad science in that way, because no new phenomena had actually been observed/measured

That is an interesting observation. Would you be willing to guess specifically why there was such a long gap between the LHC and the previous collider? The answer is already in your comment.

Nobody forces academics to work for low pay. I mean, that's what I'm doing right now and I certainly don't feel exploited.

> Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter).

Suppose all TeV-scale colliders had been defunded in the 90s, including the LHC. Then we would have forever been stuck with the strong suspicion that a Higgs is there, but no proof, along with untested but plausible-sounding well-motivated speculation over what else could be. Is that not worse than actually knowing?

While it is true that nobody forces you to work for low pay - there is definitely misleading advertising at play.

Most young people that pick science do not understand the "real" rules of science, and what it takes to "succeed". Being a good, reliable, hard-working scientists will not ensure your success.

Whereas in many if not most jobs, being good at the job itself usually suffices.

I know what the rules are. In fact the community seems to be extremely transparent about it -- there are sites that keep track of all job offers out there for you, along with historical statistics.

The struggle for jobs is the result of the government funding structure, plus supply and demand. It's not perfect, but it's infinitely preferably to the older system, where you could reliably do science only if you had great personal wealth, or were favored by somebody who did.

First and foremost, we are talking in general, about young scientists. Do you think they all know what they sign up for? I for one, having seen many hundreds of them, I don't think so.

Then I will say, job offers are not relevant here. Who gets the job offer? What do they have to actually do in that job? Where are they going to be in 10 years? What kind of life, job security, job conditions, career advancement prospects do you have over the long term? For most scientists, these are the most nebulous concepts ever.

You may think you know what the job is - maybe, I wouldn't know.

What I do know that most people do not understand that a PI (principal investigator) at a University does nothing even remotely similar to what a postdoc (who wants to be a PI does)/ The churn and exploitation is very common.

> I certainly don't feel exploited.

Sure this is your subjective perception, maybe fuelled by the positive reinforcement that rituals of accomplishment provide and the fact that working on physics is really fun. As long as you realise that you are playing a game that is unlikely to yield any economic or societal value (in the case of high-energy physics) at any time-scale with a group of people that have been playing the same game for >20 years with no discernible progress except for the discovery of one particle predicted >40 years ago.

> Suppose all TeV-scale colliders [..]

Well this is hypothetical, but what would most likely have happened is that the ~10000+ scientists involved with the LHC and high-energy physics had gone into different subfields of physics. Hopefully the same would have happened with funding (we don't want it to go to biologists, do we?). Since high-energy physics still attracts some of the best students, this would have actually disproportionately improved human capital in other areas of physics. Whether the Higgs was there or not was never a super pressing concern anyways. Condensed matter physics, bio-physics, environmental physics, all kinds of (mostly experimental) quantum physics still have discoveries to be made with ~1-10s of million in budget. Not only that a lot of those discoveries will have society level consequences in a time frame of decades. We are in contrast very unlikely to derive any benefit from studying the energy scales high-energy physics has reached. For those reasons I think funding high-energy physics is a huge net negative to overall societal progress.

> They are paid almost nothing (a checkout clerk at a Swiss supermarket makes more than a PhD student at CERN). Then you string along a subset of those for years, exploiting them for further cheap labor, somehow making them believe they are “lucky” to get that opportunity (a postdoc at CERN pays a fraction of what you can make at Google Zurich).

That’s not unique to LHC, or CERN, or physics. It’s a general problem of academia, where PhDs and postdocs are paid a pittance compared to what they could otherwise earn in the industry. This problem is especially bad in high energy physics of course, since jobs are especially limited, and it’s the brightest people competing against each other, who could easily land jobs on Wall Street or Silicon Valley.

> Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter).

Standard Model works exceedingly well at LHC. No one was actually sure about BSM (beyond Standard Model) so there was no “promise” really. Or the promise is: we may see something interesting, or we may disprove some otherwise interesting theories.

> Defunding the super collider in the 90s in the US was probably one of the best science policy decisions they made.

Cancelling SSC was such a stupid waste of labor and money, it’s painful to see someone touting it as a triumph. Two words: defense budget. Enough said.

Disclosure: I worked for CMS for a while. (Not physically at CERN; was doing data analysis for CMS in the U.S.)

Want to throw out that PhD students at CERN & ETH/EPFL are still paid a living salary. Go outside of the ETH system and you can make a fair amount more (nothing considerable, but equivalent to what high school teacher in the states, which for a PhD student is nothing to sneeze at).

I'm pretty positive that people made firm predictions and bets that the LHC will see super partners (because otherwise the "naturalness" argument would go away). Whether it would be the MSSM or something else was up for debate, but people thought it would be more likely than not that they would see them if the Higgs was found in the predicted energy range. In any case physics departments at top universities all over the world are stacked with phenomenologists that made their careers on working out these predictions.

Anderson made an argument against the SSC https://www.the-scientist.com/opinion-old/the-case-against-t..., which I pretty much agree with. Science funding is finite and physics talent in a country as well. Many really good students are funnelled into dead end careers in high-energy physics (whether theoretical or experimental). It's just a huge waste of human potential, especially given the fact of how ruthlessly they are exploited. I know people in the field, a hiring decision between three people was recently described to me as a choice between a 'social case' and two competent workers, one of which happens to be a friend of mine.

Funnily enough lot's of institutions doing fundamental research in high energy physics either also do military research or receive military funding. Most of Witten's work for example has been funded by the Department of Energy. The whole reason CERN was build in a neutral country was because people worried that a post nuclear arms race would break out otherwise. In France one of the major institutes contributing to particle physics (Saclay Nuclear Research Centre) also developed their nuclear arsenal and is located next to a major arms manufacturers research center (Thalys).

> more likely than not

Yeah, “more likely than not” isn’t a promise. Sure, a lot of people firmly believe in their theories, so me saying “no one was actually sure” seems wrong, but I was talking about a different kind of “sure”. The community overwhelmingly agreed on SM, whereas there were huge divides on where the BSM bets were, or even on roughly the same bet, where SUSY scale lies, etc.

> Mant really good students are funneled into dead end careers in high-energy physics, ...

I was one of the funneled. We signed up because we were drawn to the fundamental questions, not because of glowing job prospects, which were largely laid out for anyone paying a little bit of attention. Cancelling things and decreasing funding certainly didn’t help, only lead to worse “exploitation” in your words.

> Funnily enough lot's of institutions doing fundamental research in high energy physics either also do military research or receive military funding.

Institutions do lots of things. Most also receive funding for medical research, so?

In general, modern day HEP in and of itself hardly contributes anything to the military sector. On the more practical side, powerful magnets, computational methods etc. should be useful in military applications, but a lot of different areas have such second-order effects. Nevertheless, I’m neither knowledgeable nor enthusiastic about killing machines, so I could be missing some obvious connections.

> Most of Witten's work for example has been funded by the Department of Energy.

Why would you put all DOE funding under defense budget? It’s not DOD. Or would you characterize all renewable energy spending as military spending too?

> In France one of the major institutes contributing to particle physics (Saclay Nuclear Research Centre) also developed their nuclear arsenal...

Particle physics has largely moved on from nuclear physics. (I know, many particle physicists are still interested in cold fusion etc.)

> Cancelling SSC was such a stupid waste of labor and money

No, it wasn't. The construction site was chosen for political reasons and the entire bidding process was beset with problems that would have made it cost even more money, run into a ton of problems (for example fire ants causing multitudes of delays and groundwater seepage making the installation of sensitive electronics probably impossible), and probably not have been finished. The administrators kind of knew it was going to cost more than they pitched even under best of circumstances and were counting on funding momentum to keep it going (well we put this much money into it so we can't just stop now).

What the US did with the SSC money was to put it into LIGO instead.

source: growing up my neighbor was this guy: https://www.npr.org/2019/05/19/723326933/billion-dollar-gamb... and he told me about these things.

> ALICE’s desaster of a analysis Pipeline

Any references to this? First I've heard of it.

You can take a look here: https://github.com/alisw/AliRoot

it is true that moving protobufs is more profitable than moving protons, but I think CERN represents a fundamentally interesting and important scientific endeavor.

Reply on news.ycombinator.com