Hacker News Reader

Comment by mabbo | Hacker News Reader

The key question that everyone, especially SpaceX, should be asking is why was it even possible to run the test while Cabada was in an unsafe place?

System and safety design- a topic that a rocketry company should be deeply familiar with- should lead us to solutions where bad things are exceedingly impossible. Where you have to be trying to cause an accident for it to happen. This should extend beyond the product being built itself, to the factory, to the R&D, to every part of the organization where danger exists.

And when failure occurs, it behooves you to shout from the rooftop that you messed up, and show the world how you're making changes so it can't happen again. Being quiet about the failure of your system makes it sound a lot like you aren't planning to make any changes to it.

It's a bad culture to start in a rocketry company.

View on HN

> The key question that everyone, especially SpaceX, should be asking is why was it even possible to run the test while Cabada was in an unsafe place?

absolutely.

lockout/tagout may seem unnecessary, but it's specifically to avoid exactly these types of situations. Not is it not possible for someone to be in an unsafe place, it's not possible for miscommunications such that it BECOMES unsafe after the fact because that lockout is with a lock no one else can unlock except the person who did it.

It obviously requires humans actually DO the procedures, but I think that's part of the point with the question.

> lockout/tagout may seem unnecessary, but it's specifically to avoid exactly these types of situations. Not is it not possible for someone to be in an unsafe place, it's not possible for miscommunications such that it BECOMES unsafe after the fact because that lockout is with a lock no one else can unlock except the person who did it.

And despite many comments in this discussion thread asserting that safety reduces progress and speed, lockout/tagout and similar procedures reduce the number of accidents and mishaps. That leads to less downtime spent picking up the body parts or cleaning a corpse out of the equipment. This is one of those "slow is smooth, smooth is fast" things. Get into a good routine with good safety measures and you can get shit done instead of cleaning up shit.

You don't even need to slow down, in some cases. I was watching a video[0] about how shinkansen trains are serviced each day, and noticed at one point[2] when the workers are ready to clean the top of the train they do a "point and call"[1] routine at a gate. Now, I am not exactly sure what the purpose is for that, because there isn't much context in the video, but I am logically assuming that they are checking that the train's overhead wires are depowered, and the gate probably only allows itself to be opened when that is the case. Even if that's not the exact reason here, I could imagine a test facility where you had occupancy sensors inside and a lockout gate that operated in this fashion, and wouldn't allow a human to be inside the room while pressure was increased in the test valve. I honestly don't get the cowboy attitude in here, we can progress to space quickly AND safely.

[0]: https://www.youtube.com/watch?v=ieRwRS0VUUM

[1]: https://en.wikipedia.org/wiki/Pointing_and_calling

[2]: https://youtu.be/ieRwRS0VUUM?t=578

There's a section in the article relating the procedure of spraying the pressurized engine with a liquid to check for bubbles. The bubbles indicate leaks. Humans go close to the engine to check for leaks. They can point and call all they want, it is inherently unsafe for them to be there. It is unclear to me how that procedure could be replaced and I won't dismiss it as cowboy attitude.

It's SpaceX, sister company of Tesla. They can have a small bot that goes around spraying and many cameras and other sensors observing. And given they're looking for potentially very small leaks from a pressurized system you wouldn't want a person physically present when the leak is discovered at high pressure anyways, not unless you want to see the effect of a very narrow high pressure stream hitting human skin (or a fast moving piece of hardware hitting their skull).

I'm sure if the tech were up to the task, they'd have a robot you could walk around with and look at things. But it'd need really good resolution and dynamic range paired with flexible and fine motor control to lean into an engine and look for tiny bubbles. This is a research field of its own. Not something they just have.

For example: Humans improve their visual perception by lightly shifting their heads to avoid reflections, gauge shadows and mitigate obstructions. If the tech can't follow, you have an inferior solution. A solution where bubbles will be missed. Just staring at a screen is not the same. Even if you can tilt the camera.

The bubble test is not carried out at full pressure. That's the point: you find the leaks before blowing the engine up because you didn't spot a leak.

The tag-out system could, for example, use a pressure relief valve. I suspect they even have one already when running the test, but if someone was in the wrong space when the relief valve wasn't in place then I assume they're not using it as a tag point.

The Shinkansen services claim zero passenger fatalities (and close to zero passenger injuries, on account of a recent mishap). If you carefully parse the language of these claims, taking note to discount those ineptly rephrasing official claims, these statistics usually do not include workers. Here's a paper discussing how those trains are cleaned: https://siliconflatirons.org/wp-content/uploads/2018/04/Trou... Exhibit 10, page 11 shows the number of employee accidents for the years 2001-2005. No year had zero. 2004 and 2005 were as high as 18 employee accidents.

Here's a fatality from just last year during tunnel construction for a maglev line: https://www.japantimes.co.jp/news/2021/10/29/national/jr-tun...

Ok but I’m talking about one specific safety effort in regards to the article not the general safety record. I’m not saying everyone has to have zero accidents, but clearly there are things we can do that aren’t having some guy spray liquid on a pressurized valve.

> lockout/tagout may seem unnecessary, but it's specifically to avoid exactly these types of situations

Thank you! I was trying desperately to remember what that system is called but deleted that sentence from my reply because I had forgotten it!

The idea that every accident can be prevented by delight is a ridiculous and utopian trope, and to demonize an entire corporate culture and group of 12k employees without any background information is the height of hubris.

Nobody would ever design a ladder that cannot be climbed without a safety harness being properly worn, but many people are injured or killed in ladder accidents at workplaces every year. In your world that's a design failure, but out here in reality it's often carelessness or sheer bad luck.

To extend that line of argument a bit, what about a more professional tool/system, e.g. cherry pickers?

Doesn’t it seem reasonable to you to design and engineer cherry pickers such that the bucket can’t be moved unless a safety harness is attached?

> Doesn’t it seem reasonable to you to design and engineer cherry pickers such that the bucket can’t be moved unless a safety harness is attached?

No - that seems unreasonable too, as there's no reasonable way for the cherry picker to know if the harness is being worn by anyone, so it's just safety theater.

Combine it with immediately firing anyone caught attaching an empty harness to a cherry picker and the cherry picker can be reasonably sure that the harness is worn by someone.

Do not make The Perfect an enemy of The Good.

Merely because some accidents still happen despite motivated and brilliant people working to prevent them does not mean that the effort should not be made.

You don't abandon the endeavor, but nor do you just throw caution to the winds.

The article had a great example of a competing space company with an evidently solid safety culture; an accident was reported, and inspection done, and zero issues were found (which is quite difficult to be good enough have an OSHA inspection find nothing of note)

>> Relativity has a strong safety record relative to others in the industry, with just one OSHA safety inspection since it was founded in 2015. The inspection, spurred by an accident, did not result in a violation.

> Do not make The Perfect an enemy of The Good.

That's all well and good, but you also can't demonize an entire culture based on unrealistic navel-gazing. It's entirely possible this was a process or design problem that could have been prevented, but it's also possible that it's not - pretending you know based on few details in a one-sided article is pretty silly.

> pretending you know based on few details in a one-sided article is pretty silly

No. It is not. There is one telling detail in the article. SpaceX was fined by OSHA. That pretty much means in itself that there were things SpaceX shoulf have done which they didn’t.

We can’t and won’t do each our own investigation. Nobody believes that the article has enough details to decide the case on its merrits. But we know that someone professional did the investigation and found the company violated the safety norms. That is what we have these agencies for.

> SpaceX was fined $18,475 by OSHA for two safety violations, one of which was rated the highest penalty level of “serious” and a maximum gravity of 10.

Though the article says "highest penalty level," according to OSHA's website had the violation been "Willful or Repeated" it would have resulted in up to a $145k fine[1].

[1] https://www.osha.gov/penalties

> SpaceX was fined by OSHA

We don't know what for, or if the fined offense would have prevented this accident.

Considering Relativity hasn't actually launched a rocket yet (they are aiming for first launch this year), they are more a prospective competitor.

It’s worth noting that SpaceX has about 20 times the number of employees as Relativity and Relativity has not yet launched anything to orbit whereas Falcon has over 180 flights. I don’t think we can objectively say the culture at Relativity is objectively safer yet. Relativity is also an extremely new company in comparison to SpaceX.

>but out here in reality

This, I expect, is exactly the kind of bad cultural attitude that @mabbo was referring to. You're advocating for accidents as being an unavoidable cost of business while preemptively hand-waving them as being the fault of careless employees or fate.

As @P5fRxh5kUvp2th points out, there are established industry safety measures to avoid exactly these kinds of accidents when working around dangerous machinery, in the lockout/tagout system.

If tests are locked out while employees are tagged into an area, you can't accidentally smack them into a coma with an over-pressurized part blowing out. By requiring tag ins to access dangerous areas, you don't have to have employees remember someone is out there, or that a test is coming, or worry about anyone getting confused on either front.

Blaming the employee for being where they shouldn't be or the other for running tests when someone was nearby is a cop out. It shouldn't even be possible for it to happen at all.

Design your systems and processes to expect that humans are fallible.

Impossible. Humans will find a way to bypass anything.

As a kid, I remember a lot of people complained about how annoying car seatbelts are, and tried to avoid them, but I haven't heard that sentiment at all in the past 20 years - everyone I know is perfectly happy with them as a safety measure. It seems to me it just required some time for the culture to shift.

> You're advocating for accidents as being an unavoidable cost of business while preemptively hand-waving them as being the fault of careless employees or fate.

This needs seconding. It's ridiculous to say that we can't demonize an entire company of 12,000 people, and in the same breath, turn around to demonize the line workers of that company.

It's no less ridiculous to instead demonize the line workers that got hurt, which seems to be both post-facto cherrypicking, and victim blaming.

You replace ladders with scaffolding and cherry pickers or similar things, though those too can have their safety issues ( https://www.youtube.com/watch?v=8YBxOA-aQQE )

I guarantee you that every job site manager in the WORLD can identify the unsafe employees, even if they're doing 'everything by the book'.

Some of it can be designed away (two-hand tools for example) but some of it is basically inherent.

Accepting at face value the claim that "every job site manager on earth" really can identify the "unsafe employees", then these managers are negligent and thus culpable for the failed outcomes.

An infinitely better course of action is to ban all known-unsafe employees from operating in the risky path. Workers who unnecessarily put themselves and everyone else in jeopardy have no business being in the path.

With that said, I'm highly skeptical about the veracity of this claim. Human behavior is fundamentally unstable, oscillating in the continuity spectrum between stable and erratic. However, the amount of variability for a given individual tends to be stable, which may be what you are saying a good and effective manager should be able to catch.

Regardless, humans are wetware and fundamentally error prone. Best to account for this element of our nature in the larger system design.

Relevant quote from the article:

> In the accident involving Cabada, OSHA’s accident investigation summary noted that “the final step in the pressure check operation, venting, was done for the first time using an automated program as opposed to the normal manual method that had been used in previous operations.”

"At 12:44 p.m. on January 18, 2022, an employee, an integration technician, was p erforming pneumatic pressure checks on a Raptor V2 Engine while at the Integrati on Stand. The final step in the pressure check operation, venting, was done for the first time using an automated program as opposed to the normal manual method that had been used in previous operations. Immediately after initiating the aut omated venting, the employee was struck by the fuel controller cover which broke free from the controller module. The controller cover had sheared at the vertic al to horizontal beveled seam, liberating the cover face from the assembly. The employee suffered a skull fracture and head trauma and was hospitalized in a com a for months."

https://www.osha.gov/pls/imis/establishment.inspection_detai...

It sounds like a number of things went wrong:

1) The automated program malfunctioned.

2) The controller cover welds failed.

3) The employee was too close.

How are you supposed to know ahead of time that a place is unsafe? Cabada should have been perfectly safe where they were, but at least one safety system failed. Ultimately you can never completely eliminate danger, at best you can reduce risk below some rational threshold for failure modes you are already aware of.

And self flagellation after an accident does nothing to improve safety. Worse, it feels like you're doing something when you're not. Further, doing so right away leads to kneejerk reactions which might feel like an improvement but which are poorly suited for actually addressing the problem. The proper approach is to methodically investigate the issue, figure out what went wrong, and make appropriate, actionable changes. There is absolutely no reason to make a public announcement at any point in the process - safety is not a PR tactic.

> How are you supposed to know ahead of time that a place is unsafe? Cabada should have been perfectly safe where they were, but at least one safety system failed.

Any potentially unsafe location should be behind a LOTO / safety interlock, and you should not be able to access it without locking it.

All locations are potentially unsafe locations. An interlock prevents people from being in a place they aren't supposed to be, it does nothing for when something goes wrong in a place you are supposed to be. You can take precautions to reduce the odds of something bad happening at the place you are supposed to be, but there's always something else that can go wrong.

> All locations are potentially unsafe locations.

You’re not even going to pretend being honest about this uh?

> An interlock prevents people from being in a place they aren't supposed to be, it does nothing for when something goes wrong in a place you are supposed to be.

What sort of nonsense is that? The point of an interlock is to keep the place where you’re supposed to be safe for as long as you’re there. And stop you from being there if it’s unsafe.

> there's always something else that can go wrong.

Yeah sure a volcano could explode from below your desk. Whatever.

> Yeah sure a volcano could explode from below your desk. Whatever.

It seems you're the one not even pretending to be honest here.

Obviously we are talking here about the sort of potentially dangerous work where there are serious risks to begin with.

> The point of an interlock is to keep the place where you’re supposed to be safe for as long as you’re there. And stop you from being there if it’s unsafe.

How would an interlock keep the place you are supposed to be safe? An interlock just stops something from running when its activated. When you shut down something to go inside for maintenance or whatever, an interlock stops someone from accidentally turning it on while you're inside. But when you're not inside and something is supposed to be running, the interlock is not going to do anything to contain a mechanical failure.

Imagine someone photographing sharks. They're in a shark cage to protect them. Compared to outside of the cage, they are relatively safe. An interlock might ensure that they are indeed inside the cage before a bucket of chum is dumped to attract sharks. But an interlock does nothing to protect them from a failure of the cage - maybe due to poor maintenance part of it rusted, or perhaps there was a defect with some of the welding to begin with - as far as anyone is aware, the photographers are in a "safe" place but really they're not.

> How are you supposed to know ahead of time that a place is unsafe?

I’m not even sure if you are asking this seriously. You make a safety assesment. People who do this kind of engineering task go through the systems methodically and they write up the dangers, hazards and the mitigations.

This is not some kind of new unheard of physics. This is not Gordon Freeman pushing an unusual crystal into an anti-mass spectrometer which results in headcrabs eating everyone who doesn’t have a crowbar. High pressure systems, cryogenics, flamable gasses and liquids all have known hazards and with proper systems and procedures they all can be mitigated.

> Cabada should have been perfectly safe where they were, but at least one safety system failed.

I’m working from a very limited set of information but the article states he should not have been where he was during a test. If that is true we have processes to mitigate that risk. (Lockout-tagout) That is pretty standard really.

> The proper approach is to methodically investigate the issue, figure out what went wrong, and make appropriate, actionable changes.

I agree with that.

> There is absolutely no reason to make a public announcement at any point in the process - safety is not a PR tactic.

Sure. But not talking about incidents is a PR tactic. Asking the grieving family to remove a picture of the hurt worker which depicts him in context of his work is a PR tactic.

> I’m not even sure if you are asking this seriously. You make a safety assesment. People who do this kind of engineering task go through the systems methodically and they write up the dangers, hazards and the mitigations.

There is a fundamental difference between "we have assessed the risk and here it is below some threshold where we are confident that no further safety measures are required" and "this place is safe, no accidents can happen here."

> Cabada, a ten-year veteran at the company, shouldn’t have been near the valve when it hit maximum pressure, according to people with knowledge of the accident, *but it ramped up faster than expected, blowing a shield off the valve* and knocking him unconscious.

A lockout would have done nothing. The test didn't start early, the test went wrong, and a place that was supposed to be safe at the time suddenly wasn't. The fact a safety shield blew off means they recognized that they went through the effort of identifying potential safety issues and implementing safeguards, but those safeguards were inadequate.

> Asking the grieving family to remove a picture of the hurt worker which depicts him in context of his work is a PR tactic.

The article doesn't say that the picture change was requested by spacex, only that it occured after the start of the family's legal proceedings against the company.

> But not talking about incidents is a PR tactic.

I mean in the sense that anything can influence public opinion, sure. But obviously what I am saying is that you're not supposed to take safety measures to improve your image in the public sphere, you take safety measures to ensure the wellbeing of your employees. A company that is serious about safety, instead of PR, will do what is effective (ie collecting evidence and making an informed, rational decision) instead of what is popular (doing something for the sake of being seen doing something which seems sensible to a person unfamiliar with the situation - like a lockout).

> There is a fundamental difference between "we have assessed the risk and here it is below some threshold where we are confident that no further safety measures are required" and "this place is safe, no accidents can happen here."

But that is not what was asked in the comment. What was asked is how you know if something is unsafe.

> A lockout would have done nothing.

You state that very confidently. Do you have insider information you are privy to?

Computers controling valves can go bad. Valves can stuck open or closed. Open wider than they were meant to. Open faster than they meant to. You take a diagram of the aparatus and go through all the possible way a valve can misbehave on all possible computer controlled valves and ask: “what would be the consequence of that?” You realise that there is a possibility that a system in question goes rapidly to a high pressure state. You then ask what harm that might cause? You realise that it might blow parts of. Or at least you are not confident that it can’t. So you ask an engineer what is their mitigation for that. If they say “oh we just know it is dangerous, we would never stand by the pipes when the test is running” then you ask them to employ a lockout system.

Look. I don’t know anything more about the test they were running than what is written in the article. You are assuming that he was in a safe place, but suddenly it was not a safe place anymore. How does that happen in your opinion? Did the pressure just suddenly teleport in? It is a ton of work to validate control software. Much easier to just stand in a place where even if the controls go haywire you will be okay.

I’m reading the same sentence differently from you. To me it means that he shouldn’t have been there during a test. (And also the test shouldn’t have blown up the valve.)

Reply on news.ycombinator.com