Boeing Starliner's flight's flaws show “fundamental problem,” NASA says
nytimes.com
nytimes.com
SpaceX are not bug free [1] and their development culture is far from being perfect [2]
[1] https://news.ycombinator.com/item?id=20877142 [2] https://news.ycombinator.com/item?id=17835760
Bug-free is a ridiculous theoretical concept, where there are humans at work, there are bugs.
As for 'perfect development culture', same thing. Ridiculous concept. There is always something to improve on.
But would I personally, with my limited knowledge about Boeing and SpaceX, but with the catastrophic fails at Boeing regarding the 737 MAX in mind, judge SpaceX the more competent company in terms of management engineering skills, efficiency, to get a product as bug-free as they can without putting money before safety concerns? OF COURSE. There is no question.
Space is hard…
Boeing used to be that culture, but has abandoned its 100 years of wisdom. Now, instead of learning from a mistake, they use marketing, blame, and PR to steer the balance sheet, not the product. They try to influence regulators instead of physics.
And holy cow, this is a long list of incidents. https://en.wikipedia.org/wiki/List_of_accidents_and_incident...
By eyeballing the list the 737 has roughly 2-3 times the incident count. Both families have ~10k units produced according to wikipedia.
Which is why in any smart organization should ironically fire someone for "choosing IBM because nobody got fired for it". Not because it is the standard but because the butt covering over actual quality is a self fufilling prophecy of trouble. Essentially choosing IBM in itself is fine so long as you do at least a surface evaluation to be sure it is actually a better choice instead of cover which subverts the actual root function.
We've sacrificed meaningful constant progress at the altar of relatively speaking small economies of scale. That's a win... for the first few years; and a loss for the hundreds thereafter.
There's nothing necessarily wrong with a segment where there's essentially one one or two providers; some things may be natural monopolies. But if so - those suppliers are essentially governments of their fiefdoms, and deserve similar restrictions, including most particularly, no incentive-distorting profit motive.
Off the top of my head I have 100s of other questions and problems with your idea. I think picking a constant number to apply throughout the entire economy is going to be fraught with edge cases.
Yes, market size matters and the problem only really applies to companies who provide strategic infrastructure and services and are large enough to dictate terms to government, to customers, and to competitors, in ways that distort and limit their market freedoms.
None of this is mysterious or hard to understand, and it's been the foundation of antitrust legislation since Standard Oil.
Europeans got seriously spooked because they were afraid that if Italian Airlines flew Italian planes and such, fragmented markets would prevent any European company from being a viable player in the long term.
So they collaborated on the Concorde, and then the Airbus A300, consolidating the European aerospace industry quite successfully.
It costs a lot of money to train a pilot to fly a new aircraft, and on top of that you have mechanics, ground handlers, air traffic controllers and many other skilled people whose operations revolve it.
Thus there is some public and industry benefit to consolidation, but at the moment the competitive landscape looks very different for widebody (B787, A350) airliners where new technology is making it to to the public as opposed to narrowbody (B737, A320) airliners where the youngest design is 35 years old.
(Some of that parallels the lack of competition in the U.S. airline market which is dominated by narrowbody airliners as opposed to international flights on which you are likely to ride on a widebody. If you're going to compress people in all three axes to get them in a 737, why bother thanking them for flying with you?)
Essentially locality means Exxon and Mobil can merge but not the only two gas stations in town. Speaking of which I remember a few gas station chain mergers resulting in closing of gas stations at an intersection not because they were redundant but laws against the same owner having separate gas stations within a certain distance.
We have since realized that Anti-Trust in the Progressive period basically was applied with very little oversight and it was not applied consistantly. It was basically a way for politicans to go after the competition of their funders. This part of Anti-Trust history is often ignored in the glorifited 'Progessives fight evil capitalists'-story that is now so popular with the modern progressives.
Under the new rules, that the judges (specifically Richard Posner) came up with you actually have to show consumers were harmed. And since then all Anti-Trust cases are judged according to that criteria. And unsurprisingly the amount of Anti-Trust conviction has gone to basically nothing.
Standard Oil being case and point, during Standard Oil the oil price consistantly went down and the quality improved. The people that ran amoke against it were not in fact the poor consumers but rather local oil companies that were getting their ass handed to them by Standard Oil. They simply could not compete with the price or the quality.
This questions seems answered by the comment it's replying to. The previous poster said:
> There's nothing necessarily wrong with a segment where there's essentially one one or two providers; some things may be natural monopolies.
So, that seems like an answer to your question: no. There are some segments where it doesn't make sense.
For very mature industries, I believe studies have found that the optimal number of competitors is usually 3 large companies. For immature industries with a lot of change and innovation, that number is generally higher, but as the industry matures, players either die off or merge. A good example of this is the American auto industry.
I think that there will/can be competitors that spring out from the lower end. The trick is to prevent the oligopolists/monopolists from squashing them through manipulation of the legal system or other underhanded tactics.
https://en.wikipedia.org/wiki/CSeries_dumping_petition_by_Bo...
Bombardier is a foreign company that is heavily subsidized by the Quebec Pension Fund.
They probably were dumping.
I don't see how anti-trust applies here.
The CSeries is a fascinating story with so many twists and turns - almost unbelievable if it was a fiction novel!
The CSeries probably did not have a viable marketing and support story against Boeing.
If you want to get wound up, then the Boeing/Embraer Civil lashup is truly anti-competitive.
https://www.reuters.com/article/us-eu-usa-aircraft-wto/wto-s...
Your example shows exactly why this idea is flawed. Three competitors is what is called an oligopoly. There is nothing healthy about that.
Specially if there is also tons of international competition as well.
When there was a real need for a new company, because of EV shift, a new one happened. If it was a real oligopoly such could not be done. In fact the Big 3 attempt at using their market power against Tesla have mostly been marketing oppertunites for Tesla.
Specially if there is also tons of international competition as well.
And Fiat is European now I think.
If the government started bidding out smaller real hardware contracts to new companies, we would see a rise in new competitors and an accompanying rise in quality and roi. To do that though lobbying would have to be severely curtailed. Probably not going to happen.
Do you have any evidence that this would actually work?
Small companies generally don't have the resources to do large project. How is a small company going to build a state-of-the-art fighter jet? It's pretty much impossible. Look at other countries to see how they do it with small companies: they don't. The European nations basically have only a few large companies, the Chinese are the same, and so are the Russians.
The reasons America had a lot more aerospace companies in decades past are 1) planes were a lot simpler, 2) they were wiling to accept higher risk back then (some test planes crashing and killing the pilot was OK), and 3) the defense budget was enormous because there was a Cold War and the US's economy was gigantic thanks to being the only industrial power left standing after WWII. #3 is probably the biggest factor, and isn't going to happen again.
As for the stuff about golden parachutes, that's endemic to US companies in general, not just contractors. I don't see how it really affects competition much since everyone does it, except that it adds costs to American companies that, for instance, Chinese companies do not have, so it's probably bad in the long term for American competitiveness, but that really doesn't affect this argument for competitiveness between US government contractors (since they don't compete with foreign companies anyway). The bit about "a nice private sector job post government retirement" is pretty much the same: it's bad for American competitiveness, but doesn't affect things much internally, except that it probably makes upstart companies more competitive as far as costs go, but since acquiring contracts is very political it actually probably hurts upstarts since they're not connected the way the incumbents are.
Also, I should add they government actually does farm out smaller contracts to smaller companies. They even have a policy I think which causes them to do this more than they probably should, in an effort to help small companies. They don't farm out fighter jets to small companies, of course, but there are a bunch of processes that cause them to farm out small things to small companies.
The other thing that's important to understand is that most small companies do NOT want government contracts, if they're not already in that business. It's not like there's small companies that would be happy to bid on these things but are being prevented. In reality, government contracting is a giant pain in the ass, so the companies that do it are ones that are interested in it, and are good at it: their processes are set up to handle that kind of business model. Bidding on contracts is a pretty arduous thing and requires a lot of specialized skill and knowledge and special staff, so companies that aren't in that sector just don't bother.
First, the thing about cars doesn't work even for individuals, because it's not comparable. Buying an all-new spacecraft or aircraft isn't anything like buying a car. Cars are engineered and designed for a market, and manufactured in quantities of tens to hundreds of thousands or more. This is more like getting an extremely custom house built for yourself (think something like a Frank Lloyd Wright house, not a subdivision house): you have to find someone who can architect it, and someone to build it. Neither of these is easy because the house is entirely unique and not even like other houses. It's not something you can just go pay someone to do and not think much about, like you can with a car: you have to be actively involved in the design and construction process.
>If the government started bidding out smaller real hardware contracts to new companies
What companies have these abilities? Would you have a super-custom mansion designed and built by some random guy you met on a street corner? Of course not. You have to find someone who can actually do the work, and you have to be careful because a bunch of people will claim they can do it, but they really can't, so you can't accept their bids, and that means you have to do a lot of work in verifying they're capable of delivering.
Building a new spacecraft or airplane isn't trivial, and isn't something some small company can just do, like you seem to believe. How is a new company going to start up that can actually do these things? The only reason SpaceX worked out is because a billionaire started it as a hobby project, and even there, it's been at it for well over a decade now. Same goes for Blue Origin: it wouldn't exist if it weren't for some billionaire who apparently had nothing better to do with his time. Even going back to the old days, this is what happened with TWA and Hughes aviation: Howard Hughes was a billionaire who inherited his fortune and got bored with oil drilling bits. You can't just award a hardware contract to some tiny company that has no ability to deliver on the contract in a reasonable timeframe.
It's not in the sweet spot of this forum, but aerospace research is incredibly capital intensive and difficult. There are economies of scale of combined R&D departments. Those labs don't last forever, but I hope my example of Bell Labs shows how productive and important they can be. You would not achieve that with Boeing divided into 10 parts.
This doesn't even go into just how impractical that is for commercial aerospace. The overhead for flying multiple types of aircraft is huge. Simulators are very expensive to produce. Pilots need large numbers of hours on different planes. It would be a maintenance nightmare since parts are tested more than anything outside of medicine and similarly stringent supply chains. The market won't support 10 commercial aircraft companies.
Things were really different back in the days of Bell Labs when those inventions were made (another example is Xerox and PARC). Companies like those invested heavily in R&D; long-term stability and profitability was more important than next-quarter returns. This just isn't the case in America any more.
I don't think large R&D orgs have a monopoly on innovation. And they aren't guaranteed to produce good results. In context, though, the US put people on the moon 50 years ago at the cost in present dollars of $.5T. Easily that much or more has probably been invested in the defense side of rocket development. I think SpaceX was applying 50 years of incremental improvements to a technology that was relatively stagnant ("it just works" conservative approach/lower government funding/general risk aversion profile). This is not to slight them at all, just putting it in perspective.
But look at the payloads. Right now it's satellites/cargo. They still haven't done a manned mission. Commercial aircraft fly thousands of miles per day and carry hundreds of people. They do this while being repaired by different teams around the world.
It's not as glamorous, but incremental updates in adhesives, radar, rivets, and other safety and repairability are big deals and R&D into safety and reliability shouldn't be discounted. That's on top of the satellite division and Phantom Works (sustained and controlled hypersonic scramjet, wings that change shape). I am not a rocket scientist, so take it all with a grain of salt.
Good. What a waste of resources.
> Commercial aircraft fly thousands of miles per day and carry hundreds of people.
Jet engines are more reliable than rocket engines, and will always be.
In your opinion, what should be the end goal worthy of resource allocation in the space sector?
Essentially capital intensive makes it harder to enter and reduces the rate of attempts while EOS affects how well they can actually compete when they get there. Take Microsoft establishing the XBox Console ve Bing. Both took large ammounts of capital to launch but one is a success while the other is generally regarded as a joke. Once they got large enough as a platform the Xbox was "on the map" and a significant segment here to stay. Bing meanwhile remains a "second best".
Other examples of EOS in action - while there are larger law firms they haven't consolidated, and most artists take commissions independently instead of forming a megaconglomerate. They don't get much of anything from it due to specialization and lack of shared demand.
One example of EOS shifting is the decline of music producers with digital music and the rise of independent artists - the higher scale lost the relative edge.
You expect the 'safe, boring' choice to be slower, more expensive, more conservative in their approach.
On the other hand, you might expect them to have expertise, and you don't expect them to cut corners on good engineering to save a few dollars.
In my experience (from software) the big incumbents have the downsides you expect, but lack the upsides.
And NASA awarded them $2 billion more than SpaceX. Boeing is just a bloated, incompetent, and corrupt mess of a company.
Earlier, in 2011, Lockheed beat Northrop Grumman for the bigger Orion spacecraft.
Haven't read the book but that kinda sounds like SpaceX today.
And really SpaceX had a general experience edge in terms of coming off of designing a lot of wholly new and successfully working non-human rated space gear.
Boeing has experience in maintaining working space gear and process, but it's not the same scale of new product and process.
I think a major part of this, as applied to SpaceX and Boeing, is historically NASA micromanaged their contractors on cost+ based contracts. That is where companies like Boeing thrived, tell Boeing exactly what to do, they can do it and at the same time they can jack up the costs so cost+ is as lucrative as possible.
Now that contracting has changed and NASA no longer micromanages but just give the contractors top down goals within a given budget, we see how bad legacy contractors like Boeing are when not micromanaged, and given budgets, rather than cost+ contracts.
I feel like, maybe, some baseline QA would be a good idea for a multi-million dollar rocket. If they were able to find this bug in the middle of the mission, then presumably with some more robust testing tools, it could have been caught in a relatively short time?
I wonder what the possible justification would be for skipping testing. As a cost-saving measure for getting an MVP or website change pushed out the door, I understand -- as the foundational control software for a difficult-to-replace, very costly hardware artifact, it seems like the very definition of penny-wise, pound-foolish.
>Mr. Loverro acknowledged that NASA failed to identify the weaknesses in Boeing’s work. “Our NASA oversight was insufficient,”
[1] https://snebulos.mit.edu/projects/reference/NASA-Generic/NAS...
[2] https://snebulos.mit.edu/projects/reference/NASA-Generic/NAS...
Software is hard. Embedded software doubly so, but... Come on... This should not be happening to Boeing.
The thruster map error should have been detected in simulation. If it wasn't, the simulator is also faulty.
I'll assume the thruster map was only slightly wrong and that whatever simulator runs they did with it were very subtly off.
Still... What a mess...
What in the world are you talking about? There’s been no reporting of anything like that. The MAX engineers failed all by themselves. No management told them to design MCAS in a brain dead fashion. Nobody “weakened” engineering leadership.
The problem is that Boeing management never said that. Moreover, the fired CEO was an engineer.
As an engineer, he was accountable to reality. As a CEO, he was accountable to the board and the financials.
Guess which won.
Moreover, the 737 Max program ran from 2011-2015. The CEO at the time was an MBA.
Traditional engineering companies don't respect software. You can tell by the fact that they pay well below market rates.
I suspect Boeing maybe treats software the same way as they've always done, just one of many boxes that need to be ticked in a long waterfall like process. They just throw more people at it hoping it gets done on whatever time line they imagine it needs to be done. They are likely to be very conservative with respect to tools, technologies, and practices (i.e. not much has changed there in decades).
The fact that SpaceX is able to come up with a new design for Starship and can then get it and the software working for several successful hovers & hops in around 9 months, tells me they are able to iterate quickly and adapt to quite major design changes. They launch a lot of rockets and they seem to be learning a lot every time they do this. I don't see how they'd be able to do this without very solid software development practices. I'd expect lots of CI & test harnasses, probably lots of emulators and other tools, and I have a hunch they probably use a lot of static code analysis etc. to prevent preventable bugs from happening.
Any engineer or tech at SpaceX is likely worth their weight in gold at any other company due to the company's perceived high bar. Additionally, I don't hear complaints about their pay either, which is likely extremely good.
No one is locking them into cages and telling them they can't leave. I would bet most everyone who is there strongly believes in the mission.
EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.
https://www.glassdoor.com/Salary/SpaceX-Engineering-Salaries...
That's true for a lot of other companies where people here have advocated the employees would benefit from unionization.
For the record, I am conflicted on engineer unionization, and my original statement was intended to be neutral.
As to this specific example, on one hand I understand the drive that comes from having a real mission (as opposed to an MBA-concocted "mission") and how a team who share that drive can execute with amazing speed; conversely, I have seen how poorly teams of clock-punchers putting in the bare minimum can perform.
On the other hand, I have also seen "the mission" used as a means to guilt trip people into overwork or accepting substandard compensation.
Of the two, I lean towards the SpaceX approach being the better one. In the absolute I don't particularly like either of them.
> EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.
> https://www.glassdoor.com/Salary/SpaceX-Engineering-Salaries....
Those salaries look to be in line with what equivalent engineers at Raytheon, Lockheed, and Boeing are making in those same locations, maybe a bit higher. Thus, if the WLB reports from SpaceX are to be believed, their per-hour pay is much lower. It's hard to tell, though, without a proper breakdown by years of experience. Also, there are certainly teams within those companies where the engineers put in just as much time and effort as SpaceX engineers. Boeing is the only one I am aware of that has unionized engineers, though.
Not that it's impossible, but you'll need tooling that prevents you from making the mistakes you'll unavoidably make when tired.
Einstein would certainly not have solved relativity by saying "Uh I guess from 9 to 5 I will work on this with a one hour break in between and then I am heading off to the fun stuff called leisure time".
No... Only if you think about these problems day in and day out you can work at in the top percentiles and this is the kind of people that get hired there.
People want to work on cool stuff (in the sense of rockets or in the sense of making the world better, or both). They will accept lower pay (per hour) to do that. Therefore, working on cool stuff demands longer hours and lower pay compared to less glamorous things. It's not an inherent requirement of the field; it's a side effect of human preferences.
I don't want a rock star anywhere near the software that operates a complex machine where a malfunction can kill people.
It can be quite common in the space industry to load non-flight or old versions late in the AIT phase in order to test on other modules, or even just to officially close old actions. But before delivering everything we would check all the versions, and make sure that everything is as expected and most importantly, as described. If that step was screwed up, it is bad news for the QA, but that would explain why the errors could be found and fixed quickly: it meant just going through all the non-conformances and change logs and understand what was different between the version used, and the version that should have been used.
Starliner was selected over Dream Chaser for that reason. And Dream Chaser would have been a much better selection.
Boeing was also selected for Experimental Spaceplane (XSP) from DARPA and basically did nothing and dropped the program, while a company Masten Space Systems couldn't survive even while they had way more innovative technology.
This human space flight stuff has the potential to cost Boeing a huge amount of credibility, they got double the money compared to SpaceX and might end up catastrophically under-delivering. They will never again be able to argue that they should be preferred based on experience.
Of course, foreign government and airlines might very well stop doing business with them, but that'll just cause the US government to prop them up even more.
Nothing is too big to fail, because as long as it's prevented from failing, the failure continues to scale up.
Anyone who has seen a space movie or airplane pilot movie knows these guys go over checklists before doing anything major. Things are checked and double checked independently. But looks like this culture has has been lost. Clock was synced and not checked before launch. Mind boggling.
But the flight software team had very smart people (those that were not driven off by the aforementioned manager) and they did what they were supposed to do. The rush to test and fly is a management problem, not a technical one.
I suspect Boeing could use similar words to describe the year they have just experienced.
Ada is still widely used and having decent Ada experience will get you a job pretty quickly, especially if you know how to get things done with most of the run-time disabled.
F-22 flight controls are all Ada for example. The new training tools are C#/C++ (Unity). Inventory tracking is Java.
You don't say.
And they finally found a circumstance that they couldn’t lie and bribe their way out of.
And remember everyone... NASA gave Boeing $2 billion more than they gave SpaceX.
This seems much deeper than just not being nimble.
>This seems much deeper than just not being nimble.
Yep, it goes much deeper than just Boeing.
I saw this argument in a lot of places. Comparisons like this doesn't have a point without context. And with the context it's not something that you may want to suggest.
Edit: In a nutshell, the starting point is not the same. Crew Dragon is derived from the cargo Dragon created for COTS years ago. The Starliner was made from scratch just for the crew transportation program.
Boeing was awarded 50% more than SpaceX. If Boeing had to start from scratch why would NASA even bother?
The reason that Boeing gets more money is that companies could set their own price, and Boeing was very confident that they would be selected whatever the price.
And this came true, they set their price 1.5 billion higher then Sierra Nevada and got selected anyway.
Was nice knowing ya!