Quench of LHC inner triplet magnet causes a small leak with major consequences | Hacker News Reader

Quench of LHC inner triplet magnet causes a small leak with major consequences

home.cern

239 pointsuntilted166 comments

Helium leaks are a nightmare. During my PhD I worked with a self-built dilution cryostat that would often have leaks in the custom-built Indium seals. To find the leaks you'd have to pump the cryostat to high vacuum, hook up a portable mass spectrometer tuned to Helium to the pump circuit and then spray different parts of the cryostat with Helium from a regular gas canister. The He would then get sucked in through the leak and show up on the mass spectrometer, which was coupled to a loudspeaker so it would cause a sound whose pitch increased with the measured He density. Once you found the leak you had to vent the entire system, remove the faulty seal and replace it with a new one. All seals were handmade, i.e. you took a small filament of Indium, placed it between the flange and the housing (after carefully cleaning everything with Acetone) and carefully screwed it shut, turning each screw only a tiny bit at each turn and going around all the screws until you could see the Indium squeeze out of the edges.

Even worse, some leaks would only show up when the system got cooled down to liquid Helium temperature. When that happened you were out of luck as you can't cool the system down to 4K before spraying it with Helium, so you had to just guess where the leak might be and replace all seals in that area until you found the right one. Going even deeper in temperature would eventually turn the He4 suprafluid, which means that it loses all internal friction. In that state it would squeeze through even the tiniest molecular cracks, so again if that happened you just had to redo all the seals and hope they would hold.

That's sounds horrific. It makes me wonder what proportion of time you'd spend massaging and debugging your lab equipment vs. actually using it?

For an experimental physicist you basically debug until you get the data for your paper or until you find your sample is bad. So you’re debugging 99% of the time.

In many fields it’s rare to build new instrumentation. You fab with well known techniques with one subtle modification, so it’s not even clear what part of your time isn’t spent debugging.

It's like a high-end fighter jet. An F-22 needs 43 man-hours of maintenance per flight hour.

And that does not include the problems with electric wires. We had an unofficial rule: after half an hour debugging the electric wires just change all of them. I still have nightmares about BNC connectors.

That’s a pretty terrible rule. Unless you’re rolling over them with chair wheels, a bad BNC cable should be easy to spot by visual inspection. In my 20 years of instrumentation work it’s rarely the cable.

It was a students' lab, where each device is connected and disconnected 2 or 3 times per day, 5 days per week. The wires get a lot of use and abuse.

Usually the problem is the tip that got slightly loose and the signal is now intermittent or null. IIRC, when we detected the wrong wire we had to put it in a box and the guy in charge of the lab equipment would fix it later.

To that rule, my team added another: when throwing away the cable, cut it in half. So nobody scrounges a cable out of the garbage can only to waste more time debugging a bad cable.

I try to do this with every cable I throw out, and encourage anyone to do the same. Flaky HDMI especially.

Basically 2-3 years of getting everything to work and testing each component, than 2-4 weeks of good measurements that comprise the main results of the PhD thesis.

Shame they don’t give you any extra certification or recognition for more than two years of learning about the care and maintenance of cryogenic high vacuum systems… that’s a whole persons job in some places!

I’m not looking forward to thermal vacuum testing for my robots… because I don’t think there’s a big enough chamber in Australia… option one is (assuming funding) building a thermal vacuum chamber that can get down to GEO level vacuum with all the fun a large volume vacuum system entails… option two is shipping the entire thing to a friendly country with larger chambers (ESA in EU or NASA in the USA) but then there’s the space technology export paperwork … and I don’t know which would be worse.

If you’ve got an apparatus with 20 parts (a low estimate) and they’re all independently 95% reliable (a high estimate), it’s only gonna be in a usable state 36% of the time.

This is a more extreme description of why I hate plumbing.

As someone who is currently in a game of cat and mouse with an intermittent drip from underneath the kitchen sink, this was exactly the first thing that came to my mind also!

I feel like I would just replace all of it than play cat and mouse. Surely it would be cheaper in time, and low cost until you figure out you have to replace the whole sink.

Graduate and postdoc labor is cheap, pressure vessels are not.

Which is kinda funny since plumber labor is expensive, and new pipes under the sink are cheap.

The plumbers made better life choices :-)

The grad students and postdocs are sacrificed... for science!

Even a new vessel need new plumbing to connect it to the vacuum pump (or pumps, sometimes you need one to to go from ambient pressure to low pressure and another to go from low pressure to very low pressure) and the pressure measuring device(s) and also sealing the holes where the wires go (I guess they have some sensor inside the vessel connected to things that are outside). So a brand new vessel means restarting all the seals from zero.

i believe the comment you are responding was talking about actual home sink in GP comment, not the LHC =)

I have replaced the p-trap under sink already, but your comment actually got me thinking maybe it's a leak around the caulk seal where the sink joins the counter or faucets!

No plumbing adventure is complete without 6 trips to your local hardware store, thinking each one is surely the last.

My family (avid do-it-yourselfers, who renovated each house we lived in) used to score project size by trips to the hardware store, both proposed/planned/expected, and also actual. :P

I had a bathroom sink where we put in a new faucet that had a pinhole leak in the tubing which you couldn't see when installed. Everything would look good, but the water would come out and drip down the drain pipe.

I recently played this same cat and mouse game with no success, so I built a better mouse trap using a disposable aluminum foil cooking pan and a Moen water leak sensor. I put the drip pan on a steep angle and bent/reshaped it to collect water at a single point where I placed the sensor. I wrapped the sensor in a small piece of paper towel so that even a single drop of water would be absorbed by the paper towel and trigger the sensor. I tested it with a single drop of water from a syringe, replaced the paper towel to reset the trap, and eventually I got a notification in real time on my phone as it dripped. Over kill? Yes. But it lead to me finally finding the issue.

Nuclear reactors offer all the fun of leak tracing and invisible cracks with the added bonus that the fluids involved are extremely radioactive and contaminate everything they touch. This is why all the "just mass produce small reactors" and "just try thorium / molten salt" stuff hasn't taken off, and may never: all the beautiful theory disintegrates into man-years of laborious leak testing.

LWR coolant is not that bad. They keep the water very clean with ion exchange resins because otherwise you can have problems with corrosion. Workers sometimes float in the water during refueling where they flood the area about the reactor and spent fuel pond and open the lid of the reactor vessel.

Leaks in liquid metal fast reactors are much more obnoxious but still manageable. Sodium catching on fire when it hits air frequently isn't as bad as it sounds (a "pool fire" isn't particularly hot or dangerous but a "spray fire" can be) but it is important to catch sodium leaks quickly without false alarms and many development projects had trouble with that.

https://en.wikipedia.org/wiki/Monju_Nuclear_Power_Plant

I always enjoy the handwaving by nuclear enthusiasts. These are some of the most difficult engineering systems routinely built and operated by mankind, where a coolant loss accident doesn't just destroy the reactor but can turn the facility into a multi-billion dollar cleanup effort in minutes.

Monju was a poorly designed machine which was particularly badly run. I recently found a book they wrote about it and when I looked at the plans I thought "I can't believe they built that in an earthquake zone". I don't know if anyone is ever going to build another loop-type LMFBR as many of the problems you can have with a loop-type can't happen in pool-type reactors.

What was most shocking about the fire at Monju wasn't that it happened but that they tried to cover it up. Neither that nor the incident where they dropped the refueling machine into the reactor vessel were dangerous to people off site but the latter sure convinced everyone they didn't know what they were doing.

Contrast that to Superphenix where the fuel transfer drum failed and they struggled with a steam turbine system that was procured under corrupt contracts but overall had a good operational record. Or the three pre-1970s cases where there was significant fuel damage in the US but they found that a core melt in a LFMBR isn't as bad as it sounds because the iodine (most dangerous radioactive element in the fission products) reacts with the sodium and the NaI dissolves in the sodium so it doesn't go anywhere. Or EBR-II and the FFTF which performed flawlessly, or the highly successful fast reactors in Russia.

I would liken this to airplanes. They are similarly difficult to engineer and maintain, they can also go very very badly in mere minutes if operated incorrectly, they also have very costly recovery procedures if they crash somewhere populated.

For both systems there have been catastrophic accidents that were national tragedies, from which we learned a lot. "The rules of aviation are written in blood," and that is also true for reactors. Past failures like the one you linked don't mean that reactors are unsafe; if anything, because we had that failure to learn from, reactors are now much safer. If we had a volatile technology like nuclear reactors and none of them had ever had any accidents, then I would be very hesitant to have one in my home town since it could very well be the first to ever blow up.

I think the reason they have seen a lot of success where nuclear reactors haven't is simply up to the fact that they are both cheaper to build and that the public can very directly see where the benefit to them comes from, whereas nuclear power is abstract and indistinguishable from coal to the average person.

I agree with your analogy to an extent, but I think it's got a very significant flaw.

The consequences of aeroplane disaster are localised in space (to the immediate vicinity of the vehicle and whatever it crashes into) and time (once the crash has happened, it is more or less over).

Nuclear disasters can have global effects, and can last for decades.

I'm cautiously pro-nuclear in theory, but am awestruck by the incredible forces unleashed and the extreme impact they can have when mishandled. It's far more potent than anything merely mechanical.

"Construction started in 1986". Is this an example of an accident that would not occur today due to eg better technologies and practices, or an example of something inherent to nuclear technology that can never be made safe?

Bringing up unrelated highly controversial topics is not good. It tends to derail threads and accumulate emotionally charged, poorly informed comments.

But anyway, leaks of radioactive materials are inherently much easier to detect than other kinds of leaks because they are radioactive. Every major hospital in the developed world has a radiation safety department or equivalent performing regular leak and contamination testing in association with scintigraphy and brachytherapy, but this has not prevented the widespread use of radionuclides in medicine.

The notion of "contaminat[ing] everything they touch" is also a misconception that the radiation protection community has tried to combat for decades: radiation is only a meaningful hazard insofar as it reaches levels comparable to the natural background radiation produced by 40K, 14C, and other sources pervasive in the natural environment. There is therefore a level of dilution beyond which radioactive contamination ceases to be of meaningful concern, just as is true with all other toxic substances.

Source: as a medical physicist in training, I work with a radiation safety department at a major US hospital.

No, it's because of radiophobia funded by big oil.

Seriously. It is annoying when some ancient pipe in my radiator plumbing decides to leak. This sounds orders of magnitude worse.

The things we humans put up with to try and understand the universe!

...and central air conditioning.

2 evaporator coil leaks in my 5 year old house already, another one currently suspected.

Why oh why doesn't everything use radiant by now?

One would imagine there must be some kind of paint you can paint onto the outside of a vacuum flask, and have it be sucked into any gaps and solidify there.

It seems they already make a product for that - vacuum grease. It can get sucked into even a fairly large crack, and plug it up, as long as the depth of the crack is much longer than the width, then atmospheric pressure won't provide enough force to send it deeper into the crack.

And the grease itself is designed not to evaporate into the vacuum.

Whats wrong with using that?

Just speculating, but helium molecules are pretty small, so will be able to leak through tiny gaps where the paint/grease would not penetrate.

Cryogenic temperatures maybe?

> paint onto the outside of a vacuum flask, and have it be sucked into any gaps and solidify there.

Stuff like paint will tend to be very permeable- if it relies on a solvent drying out, then obviously its permeable enough to let the solvent through. Even if it does dry to a very solid state, there will be a very very long tail of evaporation of volatiles inside the substance.

> It seems they already make a product for that - vacuum grease.

Vacuum grease isn't perfect- it's not recommended for the high end of high vacuum or for UHV. Or for single digits of kelvin.

Another big factor is that if you're doing science, you probably want to go out of your way to eliminate contamination like that. Guessing about the possible chemical interactions of a secret proprietary substance under extremely low temperature and pressure is not a fun way to get your doctorate.

Yes, a huge factor is the outgassing of materials. While just solidifying and a long time after that, it gases off solvents and other small particles that you constantly have to pump away and that raise your pressure noticeably.

In cryogenic environments this is less of an issue because things are more sticky, but in a room temperature apparatus targeting ultra high vacuum this is troublesome or impossible.

Not sure if there's vacuum grease that can work at very low temperatures, the upper stage of the cryostat would go to 300 mK and the lower stage to 20 mK, at that temperature non-metallic compounds tend to get porous. There are materials that can work in there but to my knowledge nothing that was an easy solution, at least at the time.

If you're dealing with single molecule cracks and extreme vacuum, even vacuum grease is going to offgas and contaminate things.

Hydrogen is EXTREMELY small, the temps the poster is talking about are very very low, and XHV/UHV is extremely low pressure vacuum.

Like 'single molecules we don't want rattling around is a problem', 'more vacuum than outer space vacuum', and 'oxygen long ago turned into a solid' temperatures.

[https://en.wikipedia.org/wiki/Vacuum_grease]

Those materials exist (see Apiezon vacuum grease and Lesker vacuum sealant). In less-demanding or desperate applications, they can work.

But like similar last-ditch repairs (think automobile muffler tape or radiator sealant), they only occasionally last. Usually the best thing to do is to go actually fix the problem.

The above advice assumes you're trying to maintain high vacuum (10^-5 torr or lower pressure). At low vacuum (10^-3 torr and above), that trickery can work wonders.

For a schlenk line in chemistry where you need 0.1 - 1 mbar, sure! But this is insane physics territory... Like other people said, grease isn't behaving like you think it would at millikelvin temperatures, or at pressures lower than those in the interstellar medium.

If you have a big aluminum chamber under ultra low pressure, guess what you're concentrating in there? Hydrogen and helium from the atmosphere that's seeping through your metal. Crazy stuff that's (sadly) not fixed by slapping PTFE paste or vacuum grease onto it.

There are some products but in systems with metal seals you don't use grease/sealant as as they off-gas various contaminating small molecules

https://www.tedpella.com/vacuum_html/High_Vacuum_Leak_Sealan...

Maybe we should "just" build this in space. ;-)

Space is far, far too hot at 3 Kelvin, and most of it is too high-pressure, too.

This sounds miserable!

I think I'll bookmark this comment and come back to it the next time I feel annoyed by tracking down a software bug.

Oh yes, way too familiar situation... dry cryostats were really a game changer. Apart of the incredible advantage of not having to refill them frequently and on Sunday, they are much more reliable and without any cold seal

AlphaPheonix on YouTube has an excellent and informative video of a very similar process of testing seals, though the vacuum is pulled for a different reason: https://www.youtube.com/watch?v=VD69crOFx10

Based on you description... Couldn't you just go so cold He goes superfluid so you find all the leaks right up?

Not my area, but I know He is very expensive (as is getting it cold enough to make He4) so using He4 as a leak detector is likely going to blow the entire experiment's budget.

A good friend of mine made a leak checker from scrap parts while I was working at a university lab. It was just for a UHV system (no cryo). I wish I still worked with him.

The whizz sound leak checkers make is good fun.

Ha, I also remember using a helium canister to look for leaks in a vacuum system in the lab. At least ours wasn’t at cryo temperatures though. Sounds like a real pain.

Could you hear the indium scream as it was being squeezed out?

https://youtu.be/mjiP5Q6g_aM?t=153

Was it ever an option to cycle the system with temporal atomic layer deposition chemistry to slowly build up aluminum oxide barriers?

Helium leaks are especially annoying, because they'll brick most modern electronics for a day or so. iPhones especially.

If they have a PCB oscillator, the gas will seep its way in there and cause a clock failure.

Man, so relieved it's just cooling one section that is meant by "major consequences", title made me think of the explosion during commissioning. Looks like it's only sector S78 that needs attention: https://op-webtools.web.cern.ch/vistar/vistars.php?usr=LHC2

This report is impressively detailed for being put out just two days of the incident, but I guess LHC having world class technical diagnostics teams is not too surprising.

When I was visiting CERN, the whole accelerator was in maintenance mode. We have seen a detector (IIRC it was ATLAS) which was disassembled in layers, so we were able to see almost all of it.

Our guide explained that whole schedule for maintenance and operation is set in stone and whole schedule is planned almost to hour resolution for the next six months or so. Also it's worth noting that, there is periodic maintenance that needs to be done, and that's also planned in advance.

Because of this stringent requirements, "several weeks" of shift is indeed a major problem, and by several I guess they are looking to ~8 weeks, if not more.

My uncle, who is not a physicist, but "one smart mf", was one of the heads of the beam instrumentation team at Fermilab, and spent a large part of his career designing, building and maintaining detectors. It utterly blows my mind.

The "major consequences" seem to be for the schedule:

> This incident will probably have a great impact on the LHC schedule, with machine operation unlikely to resume for at least several weeks.

I also read it instantly in case they had a black hole on their hands, even though I'd expect us all to be dead almost instantly if any real "major consequence" event truly happened in the accelerators.

You can't create a black hole in LHC. Although sometimes when I read social media I wish they could.

Even if the tenth digit of pi can't be a six, it still might be a six, namely when you don't know it can't be a six because it is a three. Uncertainty is different from possibility.

This is just my personal opinion, but the mechanism that creates particles and black holes is the same. Meaning without throwing incredible mass at it, you won’t make a black hole. You can’t. You’ll make a particle or break apart one. Which is in fact what happens. There are also many natural events that resemble what the LHC does, at even greater energies and much greater mass, and yet no black holes. A black hole is dangerous because it’s a planet worth of mass in compact singularity. If it has no such mass it can’t do anything.

Putting this on a slightly stronger footing:

Black holes don't have to be enormous. We focus on the enormous ones because we know how they form from stars, but there's nothing inherent in black hole math to prevent tiny ones.

You can, in theory, create ones from energy, because mass and energy are the same. It's not impossible to create a tiny black hole in a collider.

Just not the LHC. It doesn't have anywhere near enough energy, even for the tiniest black hole. And even if there were some unexpected physics that did let a microscopic black hole form, it would instantly vaporize in a burst of Hawking radiation. (The smaller a black hole is, the faster it evaporates.)

If somehow all of that were wrong, and the LHC did create a tiny black hole with a lifetime longer than a yoctosecond, it could grow and become a real problem very quickly. But if you're inventing that much physics, you might as well just worry about the appearance of a mega-space-goat that eats the Earth, because it's equivalent levels of guessing.

I found a more detailed analysis here:

https://youtube.com/watch?v=Dp2C4J2yMe4

Apparently the probability is indeed very low.

"at least several weeks" is technically correct but I think a bit of an understatement. A sector takes about 3-4 weeks to warm up and 4-5 weeks to cool down. Then it needs to undergo powering tests and probably some "training" quenches.

Since the winter shutdown is scheduled for end of October (thanks to the energy crisis), there's a good chance we are finished with proton collisions for the year. If the leak can get fixed and the sector cooled down by mid/late September, we might have time for the heavy ion run. Last year's ion run was cancelled due to shortening the year, so 2 years in a row would not be great.

Is there any plan to migrate to higher temperature superconductors which might not require such low temperatures?

They introduce their own problems. They're typically brittle ceramics, making them hard to work with. They're more expensive to manufacture. And they have a lower critical current density (i.e. there's less current they can carry before losing superconductivity).

The Future Circular Collider is looking to switch to niobium-tin from LHC's niobium titanium. https://en.wikipedia.org/wiki/Future_Circular_Collider Unfortunately they'll still run at liquid helium temperatures-- while advanced superconductors have higher critical temperatures, the critical field (magnet strength) still goes higher as the magnet gets colder, and that's the figure of merit for collider designers. http://hyperphysics.phy-astr.gsu.edu/hbase/Solids/scbc.html

This is why you see new fusion reactor designs like the SPARC which use HTS superconductors throughout still use mildly exotic cryocoolants like liquid hydrogen-- not as expensive as liquid helium, but still better performing than liquid nitrogen. (Not to mention that liquid nitrogen is annoying in nuclear applications: it's easily activated by neutron radiation and deposits monoatomic carbon dust through your cryocooler circuit)

They would have written "Unforeseen Consequences" in that case ;-)

(https://developer.valvesoftware.com/wiki/Unforeseen_Conseque...)

A black hole is not unforseen, they don't call the place Black Mesa for nothing. But a resonant cascade, on the other hand...

> I also read it instantly in case they had a black hole on their hands, even though I'd expect us all to be dead almost instantly if any real "major consequence" event truly happened in the accelerators.

Idk why people think this. Black holes are not magic. They have exactly as much gravity as the mass had which collapsed into them.

As you go about your life right now the gravity of Switzerland is pulling you towards it a tiny bit. If due to some crazy event the mass of the whole country would collapse into a black hole that black hole would still tug you the same way and the same amount.

I’m not a physicist, but it seems like they’re a little magic, in that they’re so far from our normal experience, since gravity is proportional to the inverse square of the distance, and the distance to their center of mass is relatively very small, so you can get an extremely strong pull relative to a normal object of the same mass. It seems like even a very small black hole falling onto things could snowball, unless there’s something that counters that? Do black holes lose material/get less dense/evaporate under certain conditions?

Yes, black holes are theorized to evaporate due to Hawking radiation [0].

[0] https://en.wikipedia.org/wiki/Hawking_radiation

> Do black holes lose material/get less dense/evaporate under certain conditions?

They radiate something similar to heat (with wild possibilities for temperature).

>I'd expect us all to be dead almost instantly

Not an expert at all but from what I've read you'd expect wrong. Even if they could create a black hole out of the Matterhorn it would be so small, less than a hydrogen atom, that it doesn't interact much with anything.

"Even though a microscopic black hole might contain the mass of a mountain, it would experience almost no friction as it passed through regular matter. It would fall through regular material as if it wasn’t there." [0]

So the worst thing would be the missing Matterhorn which pretty sure would upset the Swiss.

[0]: https://www.universetoday.com/1930/are-microscopic-black-hol...

So… would it fall to the center of the Earth? At that point could it start consuming matter? I have to admit I’m very ignorant of how a small black hole could grow (or not).

Not saying I believe the LHC can make those or anything, just intrigued by the questions.

It’s so small at that point it begins to act a little like a white hole (Hawking radiation iirc?) and evaporates quickly. It’s surprisingly difficult for matter to cross an event horizon.

Edit-actually having a small black hole would unlock some truly sci-fi sounding tech. Like the fastest interstellar travel, seemingly unlimited power, etc.

If they manage to create one with almost no velocity, it would keep falling through the Earth, down, up and down again, for either something between atoseconds or centuries, until it either explodes or hits enough matter to get big enough to matter.

Or, more realistically (as realistic as we can be talking about the LHC creating black holes, AKA, not realistic at all), it will be launched into space in some random direction.

You're leaving out evaporation. A black hole with the mass of a mountain would evaporate on the order of a second.

That would also release energy the equivalent of ~20M megatons of TNT, which is half a million times that of Tsar Bomba and around that of the KT event. The Swiss wouldn't be too upset about anything.

There was some hope at one point that something like the LHC might produce black holes, but not the type you're thinking of as massive gravity wells that form when stars collapse. It has been postulated that the early universe in the immediate aftermath of the Big Bang may have contained primordial black holes, and any that were large enough to still exist today might be a possible source of dark matter. Any formed in a particle collider would be particle mass, not star mass, and have the gravitational attraction of particles, and evaporate almost instantly due to Hawking radiation. They'd have been harmless, but if it had been possible, it would at least confirm they actually can exist and observing the decay would have been our only realistic hope of observing Hawking radiation likely at least in this century.

Vistar is so cool :)

Also, here's a public grafana I just found with temperature charts in S78 and a bunch of other cryo data: https://dash.web.cern.ch/d/iDRuWWHGz/sector-78-trend

Not that I can do anything very interesting with that particular bit of information, but I'm always happy to see open data. We could really use more of that in public science, and CERN is really doing a great job

It is amazing that the beam was "dumped" just milliseconds before the magnet quenched! If that hadn't happened then the beam could have crashed into the magnet and caused a lot more damage.

Dumping the beam means diverting it out of the main LHC ring and crashing it into a specially designed buffer (I think it is a lump of steel or something). So cool to see this all happening automatically.

It hadn't occurred to me that there would be significant energy contained in a beam of sub-atomic particles. Something like 100kg TNT equivalent (~500MJ)? Wow.

That's crazy. I never would've thought that either. These small facts which you can barely read in "mainstream media" are what makes the LHC absolutely fascinating and impressive.

I was always like "Yeah that's just a small beam and the magnets are to navigate it in circles". In retrospect it does make sense now why people were concerned about the collision of such beams.

Also, when you really think about it. Why would you need 27 kilometre long circle if you weren't dealing with some absolutely massive amounts of energy. Couldn't you do it with lot less?

Indeed. It’s all about the curvature. Even with all those incredibly fancy superconducting magnets you just can’t force protons this fast onto a circular path any smaller than this when they very much would like to go straight.

Yeah, it has the energy of a semi truck going at a few hundred mph but in something that is about 1nanogram of mass (about that of the average human cell).

How much of the energy is in the moving electrons/protons, and how much energy is stored in the superconducting magnetic fields?

All of it in the relativistic kinetic energy of the beam. The energy in the superconductor magnets is a whole chapter of its own; their energy dissipating into heat is what makes the Helium boil off during a quench.

Also static magnetic fields don't "do" work, by that I mean, that you can not extract energy from a magnetic field by sending particles through it: The Lorentz force is perpendicular to both the movement of the particle through the field, and perpendicular to the field itself. Hence taking the inner=scalar=dot product of the force vectors and trajectories they come out as 0, i.e. no work is done.

You need dynamic magnetic fields to do work.

There's a great article on it here:

https://www.symmetrymagazine.org/article/parking-the-lhc-pro...

"Enough stored energy to melt a ton of copper"

"The beam dump is a solid graphite cylinder 8 meters long and under a meter in diameter. It’s wrapped in a stainless-steel case, filled with nitrogen gas, and surrounded by iron and concrete shielding."

Does the beam dumps make the meterial they were dumped into radioactive? What is that process like?

As discussed in one of the decommissioning articles someone else posted, it does make the beam dumb radioactive! I hadn't thought about it but it makes sense with how much energy you are pumping into not that many atoms in the end, and when you do that, you tend to get radioactive atoms.

Not just any energy but ultrarelativistic protons! That’s going to result in all sorts of interesting daughter nuclei when they slam into the atoms in the buffer.

The protons that are hitting the dump are at energy far beyond the binding energy of a nucleus. When they hit nuclei there, they shatter them. A shower of progressively less energetic particles forms, including large numbers of newly freed neutrons.

There are accelerator-driven fission reactor ideas that would use ~1 GeV protons (much less energetic than the protons here) to produce neutrons to drive a subcritical target. These might be useful to destroy certain nuclear waste isotopes.

If like me, you want to see what it looks like : https://home.cern/news/news/accelerators/autopsy-lhc-beam-du...

The amount of materials science knowledge so casually on display is amazing.

So now I'm reading:

https://iopscience.iop.org/article/10.1088/1748-0221/16/11/P...

And I want microphones, I want to hear the thing ring with that 2.3kHz note. I want to feel the 27Hz wiggle and the 196Hz thump. I want to get the Slow-Mo guys in there to place their camera, and watch the thing jump when a beam hits it.

The amount of energy in that thing just defies intuitive understanding from reading a paper, I have to use other senses.

According to the link below its an 8m long steel tube filled with various types of graphite.

I was there when there was a (much bigger) quench during commissioning of the LHC [0][1]. It was also at the focusing quadrupole modules - maybe the same ones?

I could make a joke about how it's Fermilab's vengeance for not having their own accelerator, but that would be dishonest: these focusing modules are the most complex ones of the whole ring.

[0]http://stephatcern.blogspot.com/2008/12/photos-of-lhc-damage... [1]https://home.cern/news/press-release/cern/cern-releases-anal...

The 2008 quench happened at the other side of the ring. (Sector 3-4 then, sector 7-8 now.) The physics run was going very smoothly up to now [0]. Last year part of the machine (RF) had to be brought to room temperature as well, after a cooling tower fault, and there was no beam for four weeks. I get the impression that this will take longer, but I hope not by much.

[0]https://atlas.web.cern.ch/Atlas/GROUPS/DATAPREPARATION/Publi...

For major click-bait for Half-Life 1 players, I propose the title to be prefixed with "UNFORESEEN CONSEQUENCES: ".

I never thought I'd see a resonance cascade...

Gordon! Get away from the beam!!!!

This report really has disturbingly strong Half-Life 1 vibes for me.

"It's not... it's not shutting down! Aah-- oh wait, it shut down. All good."

The major consequence here is a few weeks of repair time, there wouldn't be much gameplay in Half-Life 1 if that were it :)

Just reading that gives you an inkling of an idea what kind of amazing feat of engineering the LHC is. I really hope governments keep funding basic research like that because it teaches us not only about nature but also how to built those incredible machines.

Partially related recommendation for the german-speaking crowd here:

The Raumzeit podcast released the second episode of a longer series about the cern. In the frist one he discussed the history and success of the cern with the leader of the experimental physics department and in the second one he talks to the guy who is in charge of operations of the proton synchrotron about the actual accelerators.

https://raumzeit-podcast.de/2023/07/05/rz111-cern-geschichte...

https://raumzeit-podcast.de/2023/07/19/rz112-cern-die-beschl...

This is a very nicely written, clear article. This is what good science communication looks like.

The inner tube is hard vacuum? Does freeze over risk rupture and also incur re evacuation and time to get those last pesky digits of nothing?

you mean "consequenches"

Not too bad. The 2008 magnet failure took two years to repair, with over 50 magnets affected.

"Major consequences", and here I was already think a beam of protons flying at near the speed of light was let loose towards the city of Geneva.

But no, just a section of the accelerator that needs fixing.

Bad time to be an LHC grad student.

Quench

Nice verb

Unforeseen consequences

Don't forget your crowbar

Tiny black holes that are going to swallow the earth in the coming hours?

Quenching a magnet is the result of poor planning and poor maintenance. The only time I ever hear about magnets quenching is at CERN. I remember they quenched ~40 at once when they first turned it on, a multi-million dollar error. What the hell are they up to over there!?

Do we know if this could have an impact on the local population? Also is there any research on the effect of the LHC on people living in Geneva and around?

I think they'll be OK: https://www.hasthelargehadroncolliderdestroyedtheworldyet.co...

> During such a quench, the liquid helium in the magnet warms up and turns into a gas that is recovered by the cryogenic system to be re-liquified, ready to cool down the magnets again.

They're capturing the helium, so it isn't released usually. Not sure what they do with the nitrogen, I'd guess that is released as it is much cheaper. But in any case the amounts involved in such a quench are only a danger in the immediate vicinity, not outside. So if you're in the same room as a quenching magnet and the room isn't sufficiently sized or ventilated, there could be danger. But that's about the limit to it.

Took a tour in local university physics lab dealing with helium. My takeaway is that liquid nitrogen is essentially free. And used to protect or keep helium parts colder. So anyway in process you have so much of it that simply having it boil away or be discarded is not big deal.

A few years ago, while doing a physics lab for students in the university, we needed ice and liquid nitrogen. It was weird that is was easier to get some liquid nitrogen[1] than ice cubes[2]. IIRC we asked for the official price of liquid nitrogen, and it was similar to the price of soft drink like coke.

[1] Because it was used regularly in a few of the research labs. Just get a dewar flask that is not made of glass, and ask nicely.

[2] No refrigerator in the lab, the cafeteria didn't have any, most refrigerators had no ice cubes or a huge ice block where there use to be the ice cubes shelf.

It's comparatively free; the air is nearly 80% nitrogen. The equipment to produce liquid nitrogen has a cost and uses a fair amount of electricity. Labs that use a lot of liquid nitrogen may produce their own, but often it's produced industrially in large quantities and trucked to customer sites.

It has a lot of uses, from dermatology to machine shops.

Severe mutations and deformities.

Brain damage for the schedule programmer.

Uh, I guess some of the locals will get second-hand bummed-outedness from their acquaintances at CERN?

Did you even click on the link dude? The picture showing frost on the outside of a metal tube in a sealed tunnel ~100 m under ground is the total extent of the exterior effects...

It is a small leak of helium that disables a refrigeration unit.

Read the article. It was a >local< event at one of the magnets which caused the whole magnet to freeze over. The "consequences" are, that the whole accelerator needs to halt operations to repair the element. That takes quite a while and will postpone the schedule..

Sure but is it impossible that something would leak when injecting massive amounts of energy in a gigantic tunnel?

But what would leak? The helium? The nitrogen?

The only real danger is if you are in an enclosed space and oxygen is displaced. Nitrogen gas is about the least toxic substance imaginable and helium will literally just go away unless you make absolutely sure to trap it.

> Nitrogen gas is about the least toxic substance imaginable

Interestingly basically all gasses appear to be bad for you in some way. Perhaps the vacuum of space truly is where we belong? /s

https://en.wikipedia.org/wiki/Nitrogen_narcosis

I can't find the study right now but I think this affect is even measurable at atmospheric pressure in that if you give replace the Nitrogen with other noble gasses you can observe improvements in cognitive function.

100% of substances are bad for you under some circumstances.

You have breathed in millions of liters of nitrogen. It is as safe as any substance could possibly be.

Besides helium, what substances are you worried about leaking?

The amounts of energy injected are not that large. 1TeV, the order of magnitude, is (according to google) the motion energy of a flying mosquito. You don't even feel that landing on your body.

The special part is that this energy is put into one particle, which makes it able to achieve other interactions.

Such particles sound scary, but the only special thing about this is that we, humans, have produced that. Our earth is bombarded constantly by cosmic high-energy particles of much much higher energy [0]. I am talking about extra-galactic particles with energy levels unknown to earth.

[0] https://masterclass.icecube.wisc.edu/en/analyses/cosmic-ray-...

You only love your collider

https://www.youtube.com/watch?v=1e1eLe1ihT0

Thank you? Yes, thank you. I think.

Do we know if... the magnet heating up, and the helium leaking into the vacuum chamber... could have an impact on the local population? I think they'll be ok!

I mean worst that could happen is that it's harmful to someone standing right next to it if it fails catastrophically (no clue if that's even a realistic concern in this case). But since this happens over 50 meters below ground I'm not too worried.

Not that it was put that far below ground for safety reasons, it was mostly to save on the amount of land required and to shield the detectors from pesky background radiation (not the other way around).

> I mean worst that could happen is that it's harmful to someone standing right next to it if it fails catastrophically (no clue if that's even a realistic concern in this case).

AFAIK, these areas are restricted while the LHC is operating, so there would be nobody near it (and I believe trying to enter these areas triggers an interlock which stops the whole system before you can get near it).

If the beam shaping catastrophically fails due to the magnet failing completely, as I understand it, you can get a beam excursion from the beam track that results in a hell of a lot of irradiated earth as the beam does what beams do to everything in its way.

IIRC from my short time intersecting with the finer points of collider discussion, they locate them in such areas where generally if such things happen, it's a non-issue; but take great pains to ensure that such events don't happen, as it would require potential excavation and removal of a lot of hardware to replace the irradiated sections to restore a contamination free loop. It's also, if I recall correctly, part of why circular paths were chosen instead of things like ovals or capsules for the path shape among other factors I think like optimization for cheapest construction.

Any loss of beam containment would continue on a tangent ensuring one and only one segment of the collider would potentially have to be replaced; other shapes would end up with magnets that were relatively more catastrophic to have fail due to the excursion geometry having more opportunities to escape along a path near parallel to the track, allowing the irradiation of additional accelerator segments. Which would be a hell of a lot more expensive to remediate.

I sometimes laugh, because in the business world, people like to talk about what is 'reasonably forseeable'. I'm a bit of a stick in the mud at such times because I can generally say that exactly no one in the room is actually interested in ptobing the actual boundaries of the reasonably forseeable, as that would require doing too much math. I'm regularly right in that regard.

Noble gases are not chemically active, but some interdementional rifts might allow access of alternative life forms to our reality.

I think you're mixing things up - the incident you are referring to happened at the Black Mesa Research Facility, not at the LHC...

The LHC was a cover. Do you really think that governments honestly will spend billions for fundamental science without a back motive? It is the ideal environment to hide a secret base where they can divert money, energy, and people to.

Unbelievable the LHC is already running for 15 years! When it was just finished it was still the heyday of funny flash animations on the internet, some of course joking about the LHC ending the world:

https://www.newgrounds.com/portal/view/469806

Yes I think it's great, but still it would be good to have research to understand the effect on local population.

It is intellectually dishonest to just say it is not possible for anything to leak and have adverse effects on people without actively researching the matter.

You should really do some more extensive research about the possible harmful effects of your comments more carefully before posting.

This one is giving me a headache.

The was a small leak of helium within the instrument. Such incidents are completely unexceptional with superconducting magnets, and pose no risk to anyone outside the room where it happen, since both helium and nitrogen are about as inert as gases can get. I suspect it wouldn’t even have made the news if it hadn’t delayed the experiments.

We know what helium and nitrogen gas do and what dangers they represent. We also know how much of them are in the LHC. The main danger is that both displace oxygen, so you can suffocate e.g. if a large amount of helium is released by a quench in a small room. But this isn't any danger to the surrounding area, there just isn't enough liquid helium or nitrogen in there.

Both gases are inert, so they really don't do anything else. There isn't anything to investigate here.

It’s just helium. Have you ever bought a helium balloon, the kind that float? This is the same stuff that’s in the balloon. It’s also used as a mixture gas by deep–sea divers. It's chemically completely inert; breathing it in doesn’t have any particular effect on the body. It’s harmless, unless you fill an entire room with it so that all the oxygen is displaced.

Are people really so scientifically illiterate that they immediately assume that if anything goes wrong with a complicated machine that it will be harmful to their health?

My guess is that people imagine that the high energy particle beam transforms the helium and nitrogen used to cool the magnets in radioactive gases that may be dangerous.

But the beam never colides with the gas deposits. The helium and nitrogen are used to cool the magnets that are used to bend the beam that travels inside a vacuum tube.

[And even if the beam colides with the gas deposits, I don't expect too much radioactive waste. Each particle in the beam has a lot of energy, but there are very few. This links https://public-archive.web.cern.ch/en/lhc/Facts-en.html says "trillions" ( 10^12 ?) of protons, but a glass of water has like 10^26 protons. A nuclear power plant has much more radioactive material and during normal operation produce much more collisions that can turn container radioactive.]

I’m not sure it’s safe to make that assumption. I really think people just hear the word “leak” and assume the worst.

Don't you think the Swiss and French governments wouldn't have thought about safety almost 70 years ago?

Safety is an important part of the collaboration, which is evident when you work there.

This is evaluated by people who understand physics, chemistry, and biology, and it is the alternative science people that tend to not want to believe that and keep warning of dangers that don't exist.

It is good to always remain critical, but after a while you've used up all arguments to explain that the radiation in your microwave is not radioactive.

I guess the scary part is that they called it a leak instead of a hole. I am sure if I say there is a leak in my sock that some will start to worry.

> Don't you think the Swiss and French governments wouldn't have thought about safety almost 70 years ago?

This isn't an argument. 70 years ago France was conducting atmospheric nuclear tests that caused people and inhabited land to be exposed radioactive fallout.

As the article says, quenches sometimes happen.

Effects on the local population? You mean those who work there underground? Their voice may change pitch a little, but its only a temporary effect.

Helium is present in our atmosphere. And presumably on your birthday party when a balloon was popped.

Effects on the wider population?

A couple of delays and reorganizations at the surrounding hotels and restaurants.

Basically: the telephone will ring.

You never know, it might have been one of those time-travelling tachyons coming back to prevent its own creation (an actual conspiracy theory during the initial problems at LHC).

For anyone genuinely concerned about LHC, we've detected cosmic rays hitting our atmosphere at orders of magnitude more energy than what LHC is doing. https://en.wikipedia.org/wiki/Oh-My-God_particle

Not sure how seriously anyone ever took it, but-

https://www.lesswrong.com/posts/C2uvzYeoMkwMmscMx/hamster-in...

How does that bode for space travel and being on the moon. That's a lot of energy

A few inches of water or regolith would probably be enough for shielding.

What do you propose exactly? What should we monitor?

By the way, I worked at CERN for a few years up to the first months of operations of LHC. Furthermore, I've been living in the region since then; for several years, my home was exactly over the LHC tunnel.

I've never got a nightmare from that!

Not sure, but the engineers all came running out of the room talking like Mickey Mouse.

Reply on news.ycombinator.com