Hacker News Reader

Comment by themgt | Hacker News Reader

Boeing's explanation is that the flight automation software runs on some sort of "timer" and the timing apparently wasn't configured correctly, so Starliner thought it was at a different part of the mission and did the wrong burns.

Astonishing to me they make it sound like it runs like an independent stopwatch and not kept in sync with the actual IRL mission parameters in a more direct/continuous way. They're talking about the automation handover between the launch vehicle to the spacecraft, and on Starliner "clearly the time got messed up" ... "the spacecraft was not on the timer we expected it to be on"

View on HN

Running it on a timer actually works quite well. I don't know how this particular system actually works, but for example the Saturn V launch vehicle handled the initial navigation for several minutes from takeoff by executing previously calculated commands based on launch time, weather conditions etc.

Also, if I have understood things correctly, the lunar lander used a timer based approach to keep the rotation of the vehicle aligned with the surface of the moon as it orbited.

Since the orbital dynamics are very well understood and there really isn't anything that can affect the motion of the spacecraft other than engine burns, there doesn't seem to be any good reason to make it more complicated.

> Since the orbital dynamics are very well understood and there really isn't anything that can affect the motion of the spacecraft other than engine burns

This is wrong.

1) Atmospheric drag (unless you're in really high orbits)

2) Wind

3) Uneven gravitational field. Our planet is not a perfect sphere. The moon is even less so.

4) Solar pressure (though very minor and kinda evens out in orbit).

The unevenness of the gravitational field on Earth is for the most part ignorable. It isn't like maneuvers are precise enough that one could account for it in the calculations anyway.

Space is empty enough and engines imprecise enough that for orbital stuff you have enough margins to correctly predict and plan a trajectory ahead of time, and just do correction burns later if necessary.

On the Moon the gravitational field is indeed uneven enough to matter, but exactly one orbiter really cares about it (LRO iirc has a specific orbit where the unevenness mostly cancels out), most simply rely on occasional station keeping burns.

Which of these can't you calculate ahead of the launch? Numbers 1, 3, and 4 seem like you could calculate it ahead of time. And maybe number 2 as well, if you do it relatively close to launch time.

They use models. No model is perfect, they all have uncertainties.

Which is why they use pilots too

When in stable orbit, I buy that argument.

But many things can go slightly wrong when firing all engines fighting your way up the atmosphere.

That is true. In the Saturn V case, the guidance computer checked the actual location of the spacecraft and issued corrections that had accumulated due to precisely that.

Atlas V burn was nominal, this error didn't happen until the capsule was out of the atmosphere.

And even more things can go very wrong with more complicated control systems.

>Running it on a timer actually works quite well.

It would seem we have an existence proof now saying that the above statement is incorrect.

Let's compare the number of instances a timer has caused an issue and the number of instances it hasn't to determine the actual rate before we make a broad assumption about a field we have no experience in due to a single test case, yes?

Yeah I’m a programmer like most of us here and I’ve seen timers abused quite a lot. They sometimes are appropriate, but more often, they are good examples of an antipattern.

The wording “it would seem” should have tipped you off that no assumptions are being made, but I guess it didn’t work.

And even if we were to make a comparison as you suggest and derive some ratio, that’s not super useful without comparing it to the ratio for some alternative approach that does not use timers.

Because one specific implementation seems to have failed a test run?

A man-rated launch vehicle failing to reach orbit seems like very good reason.

The complexity of adding a remote man it the loop creates way more room for error than running a timer. They just fucked up plain and simple.

This is like saying that because the airplane crashed because the wing was incorrectly designed and popped off, everyone should fly biplanes...

> This is like saying that because the airplane crashed because the wing > was incorrectly designed and popped off, everyone should fly biplanes...

We do that. Not with the wings, but with the engines. When jetliner engines were less reliable, we required 3 or more engines on flights that included long sections without passing near an airport (mostly transoceanic flights). Now that some engines are more reliable, we have ETOPS certifications for jetliners using two high-reliablility engines. And yes, there still are two, because no matter how reliable the engines may be, one _may_ fail.

What is backing up that CST-100 timer?

https://en.wikipedia.org/wiki/ETOPS

The man-related launch vehicle worked flawlessly. This was a capsule failure. They’re built by different entities.

It did reach orbit. They're just not at the orbit they want to be at.

I thought the space shuttle was the same. They took wind measurements before launch and the ascent was hard coded based on that data.

The Space Shuttle was always hand flown, not automated.

Launch was automated. Landing was manual but not for technical reasons.

Didn't they do one automated landing? I seem to remember reading that.

They could do it but the pilots wanted something to do so they got to fly manually. The Russians did unmanned launch and landing with their space shuttle Buran.

Wind?!

Onnplanet earth, where those things are launched their is an atmosphere with wind. That wind impacts where exactly the rocket leaves the atmosphere and deploys the space ship.

This is a very disappointing approach I see on HN. Armchair analysis based on sparse information in the article assuming all kinds of things about the avionics, space hardware and rocket physics, condemning an entire team of software engineers that are working on this issue.

Think about this - if you're one of those software engineers and you read some random person on forums criticizing your work without context or thorough understanding, how would you feel?

I love HN for insight and intellectual debate, this kind of analysis does not add any insight - it is flat out dismissing and condescending.

Edit: Also, this behavior against Boeing is unwarranted because it has bias of 737 MAX issues. Completely different team, completely different problem.

Yes. For the most part, software engineering is "easy". Find and apply appropriate data structure and algorithm, done. Managing a large legacy codebase and dealing with the soft factors is the hardest part of the job.

This leads to a phenomenon where software engineers tend to come up with "one weird trick to solve the space shuttle, NASA hates him". And it's always specifically software engineers.

Generally, most people are just as competent at their job as you are at yours. If the obvious solution is not being used, there is usually a very good reason why. A lot of fields have problems that are hard for reasons of physics or other things that are not trivially gone around, and the solution is not one software engineer swooping in with a great idea that totally revolutionizes avionics.

(sometimes of course it is social factors, same as software engineering. The best way is too expensive, or management is making bad decisions, etc)

The jump to simple solutions in hindsight bothers me. Especially with software in complex safety-critical systems it's often incredibly hard to capture all the permutations that can lead to a failure mode.

I wouldn't doubt most programmers could do an failure mode effects analysis on the software and figure out they need to mitigate an event where the timer fails to sync. But how many would also capture how to mitigate that sync failure at the precise time the system lost satellite comm? Probably a heck of a lot less.

Now multiply that by the total number of software failure modes (including those latent ones we just get lucky with) and see how many get captured. There's a reason why software in these types of systems is incredibly hard to test.

This problem is software engineering, isn't it? How are you defining it?

>this behavior against Boeing is unwarranted because it has bias of 737 MAX issues. Completely different team, completely different problem.

Wrong. Do you somehow thing there's a different management team running these parts of the company? There's only one CEO at Boeing, it's only one company. And culture starts at the top.

Questioning Boeing's competence here is entirely warranted when the company has already PROVEN that it is happy to put profits ahead of safety.

I would wager that even if you find the one guy/girl who made this timer error, the decision to not use other sensors, other failsafes was probably made by a higher-up or a team.

It's usually driven by system safety analysis criteria. E.g., if a timer failure is categorized as "catastrophic" it may be required to have X amount of redundancy. The amount of redundancy defined by 'X' may be driven by customer (i.e. NASA) requirements.

Culture is culture. The 737 Max issues relate to the 787 battery issues relate to this. Is the company run by safety-first oriented aeronautical engineers or not.

"Completely different team, completely different problem"

It...seems sort of similar to me, based on the article I read. It's not similar at all in the consequences, because of the context of the flight. But in the way things went wrong.

A question that I have unanswered is, Boeing was reported as saying that if there were astronauts they would be "safe"...but would it be possible to retrieve them?

The problem is that there is ample press and experts who have scrutinized the 737-MAX issues for over 8 months now - there were multiple systematic/organizational issues and oversights all over the place - that should be duly criticized and it was, but let's learn about this other issue in a completely different division (Space), take hints and jump to conclusions about their incompetence. This is immature, condescending and disrespectful of the engineers that worked on making this happen.

SHIT happens in space flight all the time. The rate at which shit happens in space is on a few orders of magnitude different than shit that happens in commercial aerospace. Ask how many times SpaceX failed to do what they do today and it will fail more times in future, no doubt.

I also understand the company culture point - yea, company culture probably isnt the best at Boeing but they've done many many things right in past decades. They're a huge company with a lot of talented engineers (and a lot of bozos). Their CEO is a complete asshat and I am sure people inside Boeing talk about it.

I just think that there is unnecessary scrunity without data to say definitively what happened.

I don't jump to conclusions about "incompetent engineers" in either case, so it doesn't affect my opinion of the similarity.

If you are actually on the team, you shouldn't be reading HN or any other random internet forum to try and get accurate feedback on your work in the first place.

That's besides the point - I was just making a remark that how would they "feel". Not saying at all about whether they should looking around on the internet for feedback.

> Armchair analysis based on sparse information

That's internet in a nutshell, and HN is not that different despite pretensions to the contrary. What partly redeems it is the better than even chance that in posts with technical subjects, someone informed and/or competent will appear and write something worthy. You learn to filter out the rest.

Agree. HN is getting full of people with some serious chips on their shoulder. Any article with the words "Amazon", "Boeing", "Google", or "Facebook" in it will be flooded with, to be blunt, a bunch of garbage posts. Not a good insightful, intellectual discussion. Just people with massive chips on their shoulders flooding the comments with noise.

>Think about this - if you're one of those software engineers and you read some random person on forums criticizing your work without context or thorough understanding, how would you feel?

Who gives a shit how he feels? He works at a company that builds death traps, and someone at that company has built one himself. People like this should feel overpowering shame and contemplate the poor life choices that brought them to this terrible place.

Boeing is a great argument for the corporate death penalty. Its stock price should go to zero, and its assets divided between its creditors and the military pieces nationalized.

>Flight controllers recognised the problem but were unable to intervene quickly enough because the capsule was passing between satellite links.

I imagine the reason timers are used is this right here. We don't have complete coverage of the earth for satellite control, so there needs to be a timer running for when it can't phone home to stay synchronized.

> flight automation software runs on some sort of "timer"

Interesting, this might explain the failure to land the rover (vikram) on the dark side of the moon by India's recent moon mission.

For example, what if the altitude assumed by the timer is off by a few kilometers, and it starts operating in a manner which is for landing within few meters.

Landings are not generally done like that though. A landing would be basically impossible without active guidance, due to it being a far more precise maneuver than in-orbit maneuvers.

Sensors, especially redundant ones, are expensive. Ask Boeing Max engineers.

Reply on news.ycombinator.com