Hacker News Reader

Comment by tromp | Hacker News Reader

> "The most likely root cause for the booster RUD (rapid unscheduled disassembly) was determined to be filter blockage where liquid oxygen is supplied to the engines, leading to a loss of inlet pressure in engine oxidizer turbopumps that eventually resulted in one engine failing in a way that resulted in loss of the vehicle," the company stated. "SpaceX has since implemented hardware changes inside future booster oxidizer tanks to improve propellant filtration capabilities and refined operations to increase reliability."

View on HN

Why do those rockets have a fuel filter at all? I get why we have them in cars; we refuel quite often, in dirty environments and with possibly low quality fuel. For space rockets, it seems that it would be (comparatively) easy to ensure that the tanks are clean and the fuel is of high quality.

On the other hand, if there wasn't any dirt, the filters would not have clogged, so I guess it does make sense after all.

Don’t forget about all the equipment needed to fill the tanks and transport the fuel to the tanks. Probably easier to assume contamination gets introduced somewhere and just slap an in-line filter.

Imagine if the engine failed from FOD. The reaction would be "Why don't they use filters on their fuel pumps?" Whenever you make an engineering decision there are trade offs like this.

Most rockets don't have fuel filters, but this requires their tanks to be kept in a much more strict state of cleanliness, and any failure to do so would result in mission failures.

SpaceX has long had a very different idea about FOD than most other rocket companies, famously Merlin engine qualification testing contains ingesting stainless steel nuts.

That's bonkers.

What size nuts do they pass through their engine? Every engine before it goes to space has to have a nut passed through the system? Like a 12mm "go into space" grade nut or a little itty bitty nut like I might find holding down a heat sink assembly in a 1998 vintage motherboard?

I'm off to the googles to find videos...

Probably more "qualification of the design", like those bird ingestion tests they do on airliner engines. And the test probably doesn't look too interesting, the nut might not even come out the other end.

Let us know what you find. I find it not 'bonkers' but a great idea, and hope they are qualifying with the largest nuts found upstream, in 'go to space' grades.

Robustness and anti-fragility may not be so critical for an expendable vehicle that operates once for something like 12 minutes, but seems a key attribute for a refuel-able and reusable spacecraft?

Oh I agree it's not bonkers from the perspective of "wait you want to put people on that bomb and then explode it?"

You can make all sorts of rules about what's not allowed to happen, but often those things happen.

I'm just ... astonished if they've got a little practice room somewhere in their factory where they give each of their new rockets a nut to process... "Okay kid here's your graduation test!". A "we need screens to cope with big things and filters to deal with small things" is probably smart for any anything that's going to be reused.

A nut's probably in some ways easier to deal with than a blob of wd40 (in your lox tank).

Rockets: "Hey what if we strap people to a giant bomb and explode it juuust slowly enough that they're launched into space alive?"

Steam engines: "No problems, we're doing that already for centuries, just with less speed and mostly horizontally."

> I'm just ... astonished if they've got a little practice room somewhere in their factory where they give each of their new rockets a nut to process... "Okay kid here's your graduation test!". A "we need screens to cope with big things and filters to deal with small things" is probably smart for any anything that's going to be reused.

That's what their McGreggor test site in Texas is for. It's a long, boring, two hour drive from anywhere or anything.

"anti-fragility" is probably not a design goal for Merlins and Falcon.

Anti-fragility means more than robustness, it means that the system is getting stronger with each adverse event, and it is not a realistic goal that ingestion of a nut should make the engine better/stronger etc.

It is all "just" about robustness, resilience.

Of course, not for individual rockets on individual flights (but self-repairing would be way cool!).

But Anti-Fragility as you define it definitely should be a goal for their overall organization and system of building and operating the fleet.

Each issue or incident should feedback into the engineering of new units and updating of existing units so that they are improved, stronger, less likely to create issues, more able to handle issues, etc. on each iteration. This actually seems to be the case at SpaceX, including this incident.

Or, am I missing something?

Oh yes, that is an excellent observation. The components themselves cannot be antifragile, but the development process certainly can.

Thinking about that, an important feature of that antifragility is not to risk human lives during the process if that risk can be avoided. Even though space exploration is inherently risky, any actual fatality is a huge setback.

Actually I'd rather have such "bonkers" tests than the other extreme, like NASA not thinking about what could happen if pieces of foam from the external fuel tank impacted the Space Shuttle on launch until it was too late (https://en.wikipedia.org/wiki/Space_Shuttle_Columbia_disaste...).

Or not thinking that an O-ring might experience temperatures below freezing in Florida https://priceonomics.com/the-space-shuttle-challenger-explos...

The really shitty thing is they knew early on the O-rings were slipping and partially extruding themselves, which wasn't part of the original design. They shrugged and ignored this, which then opened the door for disaster when the O-rings behaved differently (and still wrong) when cold. If they had pumped the brakes when the initial deviation from the design was discovered, it wouldn't have ended in disaster.

My memory saya that some engineers objected, but they were ignored.

https://en.wikipedia.org/wiki/Roger_Boisjoly#O-ring_safety_c...

"Boisjoly wrote a memo in July 1985 to his superiors concerning the faulty design of the solid rocket boosters that, if left unaddressed, could lead to a catastrophic event during launch of a Space Shuttle. Such a catastrophic event occurred six months later resulting in the Space Shuttle Challenger disaster."

Engineers objected and were ignored for the foam strike as well. What I don't know is how many engineers object, are ignored, and everything goes fine anyway. Maybe these are the only 2 times engineers objected. Maybe they get 100 objections for every launch.

That one is a classic example of normalization of deviance. They knew there were temperature limits, but they'd pushed the boundary a bit in the past with no ill results, and so began to feel complacent about the risks of pushing them even further.

The sad part about this issue is that this was an ongoing problem. NASA 100% knew about; they just ... didn't address it.

Here's an account of just one such incident:

https://spaceflightnow.com/shuttle/sts119/090327sts27/

The tradition of passing nuts comes at least from the history of developing engines for Soviet Moon launcher, N-1. The engines used on all four launches, NK-15, were notoriously unreliable, and the suspicion was that their turbopumps, having quite tight clearances, sometimes had rotors touching walls during work. In oxygen-rich environment that led to engine fire. The engine which fixed those problems, NK-33, was working much better, and to demonstrate that it's robust for the problems like turbine misalignment, some metal parts, like nuts, were intentionally dropped into the propellant flow.

I can guarantee you that “ingesting a nut” through the engine is not part of the official engine qual program. Maybe it happened once and somehow the engine survived and they were able to ascertain what had happened. But this is not a common thing.

Edit: I will say that what actually is a test commonly done (also in industry in general) is doing a “roll test”. They put the stages on these massive rollers and slowly turn them… listening for any “clinks”. Pretty funny seeing it done.

https://www.smithsonianmag.com/air-space-magazine/is-spacex-...

> Part of the Merlin’s qualification testing involves feeding a stainless steel nut into the fuel and oxidizer lines while the engine is running—a test that would destroy most engines but leaves the Merlin running basically unhindered.

They're not running it through the engine, they're verifying that the filter is installed and functional so it doesn't get to the engine.

2012 article so it predates my time by quite a bit. Not a thing anymore though.

Things change.

Maybe it's more of a type certification/ crash test thing. Test one to failure and assume it works

Wonder how many of the nuts it took to fill up that filter? ;)

Those are big tanks.

yeah and even if starship tanks are perfectly clean you would also have to assume the tanks on the trucks delivering the fuel/oxi are also perfectly clean. As well as storage tanks, plumbing, and everything else.

Unless you filter the fuel as it's pumped into the rocket's tanks.

wow. Where I can read about those steel nuts? I wasn't able to google anything on the matter.

I was wondering about this as well. Maybe the filter isn't for dirt, but ice? (Water or solidified methane.)

Also meaningful to consider that as a vehicle meant to be reusable, it's going to be pretty much impossible to keep the tanks pristine over time unless they add a cleaning process that would be expensive and time consuming to every refurbishment (which runs counter to Starship's goal of rapid reusability). Relative to all other rockets, SpaceX's are refueled pretty often.

Better to just design with filters.

> Better to just design with filters.

I wonder how frequently the filters need to be clean/changed? After every flight? After X flight hours?

And what is involved in changing them?

I imagine, initially at least, they'll remove all the engines after every flight, and that will give them easy access to inspect/clean/replace the filters, when necessary. However, I expect the longer term goal is they don't want to have to remove the engines after every flight, just like how a commercial aircraft doesn't have its engines removed after every flight.

In my mind a rocket engine would be much more tolerant of debris since you just pipe the fuel into a giant fireball (yes I can practically see the eyes rolling, sorry! :-). For ICEs I can understand clogging tiny injectors or carbs would be a problem, do rocket engines also have injectors or other narrow parts susceptible to clogs?

The problem is that that fuel gets routed through a thin tube around the engine bell and then goes through a turbo pump before going into the engine. Both of those would have tight tolerances.

Yes, rocket engines have many small orifices to atomize and mix the propellant.

It's like a showerhead, except if the fuel side clogs, the oxidizer might eat your chamber wall.

> if the fuel side clogs, the oxidizer might eat your chamber wall

Ah, good old 'engine rich combustion'

I still remember the exhaust plume of one of the experimental Starships turning green. "Whoa, it is eating itself!" And verily it was.

yeah copper burns green and is commonly used in liners and other components. If it starts to burn, which it's not suppose to, you see green. You'll see a green flash when the merlin engines startup but that's because of the hypergolic fluids used to get the pumps running and ignition started, not copper burning. During Falcon night launches you can really see the green glow of the starter fluid on ignition.

> do rocket engines also have injectors

Yes[1]. You want the fuel and oxidizer mixed as well as possible to achieve efficient combustion.

There are also other small channels fuel has to flow through, like the ones used for regenerative cooling[2].

And not sure how well most turbopumps[3] would tolerate debris either, though that probably depends on the exact design.

There are some really simple rocket designs out there that I could imagine tolerating debris (like solid motors[4] or pressure fed hyperbolic engines[5]) but Raptor definitely doesn't fall into those categories.

[1]: https://en.wikipedia.org/wiki/Liquid-propellant_rocket#Injec...

[2]: https://en.wikipedia.org/wiki/Regenerative_cooling_(rocketry...

[3]: https://en.wikipedia.org/wiki/Turbopump

[4]: https://en.wikipedia.org/wiki/Solid-propellant_rocket

[5]: https://en.wikipedia.org/wiki/Hypergolic_propellant#Characte...

I posted upthread, but another concern is that stray particulate is an ignition source. Rocket fuel pumps are ~100,000 hp, and that energy is put into kinetic energy of the fluid flow. A stray particulate can cause the metal pump and fuel lines to burn in the presence of O2 at 300 atmospheres.

> Rocket fuel pumps are ~100,000 hp

That is wild. I need to learn more about rocket fuel pumps :-) Thank you all for these comments.

They also arent as big as you might think. Much of the engine size you see in pictures is the Nozzle

https://www.teslarati.com/wp-content/uploads/2018/08/Block-5...

Fuel pumps are one of the most important design considerations of a rocket. When people talk about a rocket's specs, it's often going to be in the approximate order of thrust, efficiency (called ISP), fuel, and then what kind of fuel pump it has, aka its cycle

https://everydayastronaut.com/rocket-engine-cycles/

Yes, they do. All of the fuel and oxidizer goes through injectors to atomize it as it is sprayed into the combustion chamber. For a Merlin 1D, that’s about 340 pounds of propellant going through the injector plate every second.

Rocket Fuel Injectors - Things Kerbal Space Program Doesn't Teach https://www.youtube.com/watch?v=aa4ATJGRqA0

The thing is the energy density, flow rate/volume, and heat flux in a raptor is just so extreme. For example, there's two turbopumps operating at around 100k HP each in a volume not much bigger than the propane tank on your grill.. and those are basically just fuel pumps, not even where the real party is (pre-burners and combustion). Anything not going according to plan like a very small piece of debris in a filter causing turbulence or a change in flow rate is almost always catastrophic.

Rockets and cars have fuel filters for different reasons.

You car has a fuel filter because dirt causes wear and tear on the engine over time.

In a rocket, a stray particulate is a potentially explosive event. the Fuel pumps are approximately 100,000 horsepower and particles in flow can have enough kinetic energy to trigger combustion with the walls of the system the fuel is flowing through. Most metals will burn as fuel in a high oxygen environment, even at standard pressures, let alone pressures 100x higher.

> Why do those rockets have a fuel filter at all?

1. Launch vibration is real. There's a reason why the Merlin 1 engine has to survive a nut being fed into the fuel and oxidizer lines while running.[1]

2. My understanding with this particular issue is that SpaceX uses autogenous pressurization. It pressurizes the LOx tank with the output from the oxygen-rich preburner. Well, that output is mostly oxygen, but it contains various hydrocarbons. Which, when combined with pure oxygen can form ice and dry ice.

It's generally not good to feed solid CO2 and H2O into a high performance turbopump that's designed to accept liquid oxygen.

---

1. https://www.smithsonianmag.com/air-space-magazine/is-spacex-...

Thos

> On the other hand, if there wasn't any dirt, the filters would not have clogged, so I guess it does make sense after all.

Things can freeze at cryogenic temperatures. Water ice, dry ice ... or other gases solidifying. There can also be FOD knocked loose by vibrations.

I wonder the kind of sensor package they fly to be able to get to this conclusion. I have a real passion for monitoring things (for me, mostly web apps and data/message flows) and this is a really interesting case.

Seems to be as simple as a bunch of pressure sensors, in this case.

True, but add telemetry, and all other sensors who have shown nominal performance throughout the flight and that gave them the information they needed with the time resolution required to see an highly energetic cascading failure from one engine to the others.

It's a very interesting space (no pun intended).

SpaceX is at least known to have some kind of high resolution/precision microphones around the structure recording constantly so that they are able to do 3D triangulation inside the vehicle.

Interesting, so they monitor the vehicle like it’s an earthquake

yes, it's also what helped them solve CRS-7

Question is: What blocked the filter? They probably have some idea, since they sound confident they know what needs to be changed.

Here is a theory https://youtu.be/ZIisBG3NV8Y?t=4445 that there was a methane leak which could have caused solid methane which could block filters.

He abandones that theory in the pinned comment.

I heard there can be issues with ice.

Reply on news.ycombinator.com