Airliner Repair, 24/7 Boeing’s traveling fix-it team (2008) | Hacker News Reader

Airliner Repair, 24/7 Boeing’s traveling fix-it team (2008)

smithsonianmag.com

153 pointstaubek97 comments

It costs $200,000 a day to have an airplane-on-ground. The AOG crew is Boeing's best engineers and mechanics. An airplane flying is earning money. An airplane on the ground loses money at a prodigious rate. Boeing's job, from design to AOG, is all about keeping the airplane in the air earning money as much as possible.

I read that Microsoft also has an "AOG" crew, to fix broken critical software systems. It's a big reason why companies by Microsoft software rather than open source. If a company's software stops working, the company stops making money, and they want it fixed ASAP.

Not just aerospace, I vividly remember the first, and last, time I saw an An-124 in person. They were hired to fly some 60 ton heavy mining equipment, a gearbox for a ore mill if I remember correctly, from Munich to some place in Asia. All in all, from hiring the Antonov to it leaving loaded for Asia, tool 3 days. That included sourcing and transportimg the equipment from the manufacturer to the airport.

Sometime time is so much money, that money is a seemingly second thought. Still the economics work out.

Side note, I was sure to see you under this submission before clicking on it! A pleasure as always!

I used to have the equivalent job for a Rock'n'Roll lighting company. I always went to work with my passport in my pocket, and more than once hit the airport for a trip that would last a week with nothing more than a toolbox and a credit card: clothes and toiletries you can buy once the show has started. That sort of service was one of the reasons that people bought this company's equipment.

There's quite high stress when having to fix something so that the 15,000 punters waiting outside can actually get to see the band they've paid for. Fun though, in retrospect

Those high-stress business situations, in which you're the one person who has to urgently make something big happen, often with uncertainty about the situation and solution, are... high-stress.

But you can play it as Astronaut Mode. Analogous to the spacecraft that will run out of air in a few hours, and you have to remain calm and sharp and fast-thinking, and in command and coordinating with others, maybe a touch of humor, to make a miracle happen. And somehow it comes together, sometimes more creatively than others. Stressful, but doable, and invigorating and rewarding.

The stress that I think kills you much harder is when when it's a year of mind-boggling pathological big-corporate dysfunction, and you just aren't allowed to make things happen. (The spacecraft has a few hours of air. Various roles in mission control are misaligned, don't know what they're doing, secretly want that mission to fail, random people getting on the radio to countermand, etc. Even Capt. "Ice" Borland will despair when they realize.)

Hence, do your own startup, or find a place you're similarly empowered to make the occasional stressful situation come together, without deadly Type BS stress.

Yep, short term stress won’t do anything to you, it will even let you have a lot of dopamine during and after the stressful situation.

Stress will kill you when it becomes chronic, that is when the stressful situation never ends.

If I understood correctly it’s because instead of getting some cortisol then flush it from your organism once everything is okay, you will add up cortisol over time and this hormone is inflammatory. So chronic stress is physically harming you.

Good point about short-term vs. long-term.

I think I conflated that in my example, with an additional factor: degree of stress. Such as I'd guess differs between when you can rise to the challenge successfully, and when all your efforts are thwarted.

I worked at a rather large bank at the start of covid and we overnighted a bit of networking kit on a first class flight when everything was more or less grounded, I think there was a 50x multiple to the cost of the item vs the cost of transport for it, time is very much money

I've seen Red Hat as potentially similar. Some customer problem comes up, and Red Hat can task engineers who are among the best people in the world at solving it. Plus all the work into having a well-engineered baseline.

(Though I've gotten the impression that IBM has at times hurt morale at Red Hat, so retention and alignment of skilled people is a concern, so hopefully IBM is improving that.)

Yep, you're not paying for a support team, you're paying for a support team that knows how to find Linus. :)

Regardless of what their impact on RedHat has been, this is old hat to IBM. They've had tiger teams like this on the mainframe and as/400 side at least for decades.

True, and I wonder how the care & feeding of blue-necktied elite paratroopers is similar and different from red-fedora'd ones.

Yup. My girlfriend's dad was an IBM mainframe troubleshooter in the 80s. IBM would tell him to be on a plane in 45 minutes, go fix a mainframe, and then go do it again. I think he burned out after a few years, or maybe he quit when they told him to cut his hair. I know I don't have the constitution to be in permanent fire-fighting mode and short notice on-call like that.

He's been a high school teacher ever since.

I did quite a bit of "be on the customer site tomorrow morning when the doors open and stay till it's fixed" over the years. Not as bad as 45 minutes, but still rough. I can't imagine how you do it logistically now, since in the age of 'if you aren't scared shitless the terrorists have won', doing a "single male booked a one-way ticket at last minute with no checked bags and a backpack full of weird electronics" is apparently a 100% guarantee of spending quality time with a TSA 'enhanced screening' these days on top of all the other miseries that travel now brings.

Back in the 70's I worked at Aph that developed handheld LED games for Mattel. Glen, the owner, would put the wire-wrapped prototypes in a Samsonite briefcase, and off the the airport. Security would wonder what the heck it was, so he'd open it up, fire it up, and play the game. It would draw a big crowd! (Back then nobody had seen anything like it.)

Another friend, Eric Engstrom, was developing a phone. The prototypes were made of modelling clay with wires and chips and LEDs. That really set the security guys off.

Booked by corporate on business/first class should help through TSA, I'd guess. Or if it's a real problem, they'd go private and bill the client.

Might be able to get the fixers TSA IDs, which could help get through security too.

Many large software and hardware vendors have something like this, sometimes multiple variants - a “fly and fix” team to do onsite troubleshooting or a critical customer program which usually provides project-management oversight to get different teams involved to help get complex problems fixed (but which might also have an explicit or just de facto ability to compel teams to prioritize certain fixes.)

So much so that I would almost count that as the definition of a serious enterprise software company.

> I read that Microsoft also has an "AOG" crew, to fix broken critical software systems.

From what I know, back in the day (more of less 10 years ago) MSFT had some Premier Field Engineer (PFE) that operated more or less like this, at least in some places in America.

For databases, I saw some of those folks fixing quite complex scenarios of corrupted log shipping, spin locks, disaster recovery, fixes of corrupted of hundreds of GBs and so on.

I did not had access but I read in some docs that MSFT used to charge almost thousands of dollars for that kind of support.

Thousands of dollars is.. not that much.

When I was a junior strategy consultant they charged 175 dollars per hour of my work. And the company managed to find clients who paid for days of such work, sometimes even with overtime. As much as I like to think of myself as a smart person, the decks that I made were solid but in my opinion nothing special, nothing really groundbreaking. Just standard strategy decks, based on "research" (google), internal analysis (data pulled from ass) or statistical model (excel with lots of assumptions). I also doubt they pushed anyone anywhere apart from making some sales.

On a side note, the rates for tax consultants were generally lower, for example a partner charged 500 per hour, but they usually couldnt charge the client for more than few hours.

So I could imagine those microsoft guys charing something like 2000 per hour, or rather something like 15 000 per day (because it sounds much better) for someone who is actually critically needed. Or maybe those guys think that others are ones charging high rates because the grass is greener on the other side

(Obviously you would take maybe 15% of your hourly rate...)

tbh sometimes I think the most important skill for any consultant is being able to find people who will pay for your services, anything else is secondary

I remember getting a call from an unnamed bank who needed to connect two Microsoft software components. They had spent tens of thousands with top-line Microsoft support and their final answer was that it was impossible.

For some reason the bank called me and my buddy. I asked why they called me. "Because you never use the word 'impossible'." I told them I wanted $20K if we got it working. Two hours later we had it working. I waited two weeks to deliver it because it needed to seem like they got their value out of us o_O

I work in video games, for one of the largest publishers in the business - I can vouch for the Xbox part of that. If you have a game out there with a critical bug that makes it crash or interrupt gameplay for hundreds of thousands of players, you get direct access to the Xbox team, all the way to people behind the OS. Most intense 3 days of my life, fixing something that ultimately ended up being an unannounced change in their API behaviour that maybe looked harmless on paper but caused our game to be unplayable for about 2 million players. Basically Microsoft gave us direct line to their best teams so we got the fix in as quickly as we could.

Yea that doesn't surprise me. This is how they have that unbeatable marketshare lead over practically anything else when it comes to gaming. They'd probably drag bill gates out of bed at 3am if it was important enough.

One of my college classes was taught by someone who worked in whatever that group was/is called. One of the best classes I took and definitely had some great stories to tell as well.

He didn't have a degree so in order to have a class that was accredited his co-worker(who was also fantastic) had to be the "professor of record" and they co-taught the class(since he actually had a masters and was "qualified" to teach).

I think this is the very same job they filmed for the criminally under-appreciated Worlds Toughest Fixes. Interested persons with an extra $2 can watch a detailed-step-by-step documentary of this very job here:

https://www.youtube.com/watch?v=D7GvfNcY70Q

That is a GREAT show, possibly the best in the genre.

Video not available

You have to pay to watch it.

How do you do that? I don't see a slot that says "Insert $2" anywhere, just a notice that "This video is unavailable."

This person made the mistake of linking directly to the episode. The playlist of the channel [0] allows a logged in person to pay for SD or HD per episode. You can access the playlist of the channel from the page linked by clicking "World's Toughest Fixes" underneath the video title.

[0] https://www.youtube.com/show/SCuCu3O5ATBHFcvv94RsRQWg?season...

All these episodes sound great!

Also available on Disney+ (at least in the UK)

Not on Disney+ in the USA, for some reason. And they HAVE National Geographic in there!

> Says Paul Amrine, quality assurance supervisor on this project, “Sometimes we go to work in the morning and end up having to ask our wives to bring us a packed suitcase.” [. . .] Man-hours, parts and resources, and a time-flow to a rock-hard completion date are calculated.

I had to do a double-take to confirm this was written in 2008, and not the 1960s.

When you're in a specialized on-call role like this you know to expect the unexpected and I surmise they're excellently compensated for this level of knowledge and expertise on the playing field.

It's the same thing with some hospital support teams I've seen in the IT world -- it bewildered me that our EMR vendor wanted to ship someone out next day to drop a replacement hard drive in just because one of the discs that's been mirrored three times over has failed. We'd live with the remaining 3 copies of the same data a little bit longer, but they saw that as a possible safety risk.

If you've ever been involved in an NTSB failure report, you do NOT want to be the person answering the "so why was the replacement drive shipped by pack mule".

While it may not be pleasant, NTSB investigations are generally meant to be blameless, no? Punishment is considered the lowest priority, reserved for specific intentional violations.

They’re usually quite blameful, just the people blamed are often dead.

But even so you don’t want to explain to the boss why you tried to save a few bucks and caused an investigation. So the expensive but safe route wins.

It's vastly cheaper for them to helicopter in (figuratively, usually...) and replace a single drive when the stakes are minimal versus deal with cascading fallout of multi-drive failures and data/system recovery.

This level of support is common for most high-end hardware systems - all the big players do this, including IBM, Dell, etc.

It seems to happen more often than not in my experience that when one drive in an array fails, one or two more go within the next week or two.

Why the double-take and reference to the 60s?

Likely the "wifely duties" aspect.

I do not see a reason to double take due to that. Asking your spouse for help seems pretty normal, and 99.9% of aircraft mechanics that travel abroad on a moment’s notice for emergency repairs being men, 90% of which are married to women, also seems normal. Even in 2023.

The 60s take is that the woman is at home, ready willing and able to pack a suitcase. In 2023 it's more likely she is at work, like everyone else.

This particular job may self-select towards a (high) single income family scenario, though. Or at least a highly flexible second income.

I like my wives at home.

Your spouse is incapable of going home and packing a suitcase to assist you simply because they also work? If you can't help out your spouse when needed, then you're not much of a spouse - regardless of your gender, income, roles, etc...

This is a ridiculous take...

Yes, people can't just leave work to rescue an incompetent spouse at a moments notice because they didn't prepare appropriately for their 24*7 on-call job.

My wife and I both lived this for a while, and we both made some pretty crazy sacrifices and trips to keep each other moving. We were equals and highly committed.

Be realistic - this isn't an every day thing clearly.

If you cannot help your partner during a time of need or emergency - then you're not really partners, are you?

Not everyone, but some can. Presumably there were other ways for non-married colleagues to figure out that situation. The author was talking about his and [some of] his colleagues’ experiences, not arguing for a position in the battle of the sexes.

The text is specifically referring to 'wife' - the underlying assumption (or fact) that the team all have wives. Not a single husband[1].

It's not about 'helping out your spouse', the 60s comment was in reference to stereotypically gendered language. It was not, and has never been, about 'helping your spouse'.

[1] Or enby partner, but the point still stands.

How do you know they weren’t all men? If that was in fact the case, would you still say the author ought to use ungendered language? Remember this is a personal experience, not a political piece.

I know it's a personal piece. I'd prefer if the author used ungendered language, but my amount of upset about that not happening is close to zero. I'm not the language police.

I do, however, very much object to folks moving goal posts on a discussion. OP commented on 60s language because it was gendered. Apply whatever value judgment you want to that statement, agree, disagree, whatever you need to do - but it was never a discussion about "helping a spouse is bad".

That was the point. Let's stop debating things that have never been said, we have Twitter for that.

This kind of commentary gets so tiresome.

The only time you'd need someone to bring you a packed suitcase is when you can't go home to get it, don't have time to buy replacements, and will be sleeping on-site. They're asking a spouse to pack things up after normal business hours, because they can't do it themselves.

It does seem unlikely that the airplane maintenance would be within reasonable driving distance of the house, the story would have been clearer and more impactful if they'd said "Fedex a suitcase overnight".

Understandable. It's a bit hyper vigilant but the sentiment is likewise understandable (even going by your estimations there could be exceptions to the norm).

That’s what I assumed but thought to be a little silly. Sure people-hours instead of man-hours and spouses instead of wives but neither of those are bad enough to make a comparison with how imbalanced things were in the 60s.

I don’t believe the original article was really a call out to the wife’s duty. Replace it with spouse and it seems reasonable enough. Too many unknowns to do a double take.

The author is also speaking from personal experience. It is totally possible that (for example) all of his experience has involved him working with other men and in all instances it was in fact, wives, who did the suitcase bringing.

I understand wanting to be inclusive but I think its a bit incorrect to police someone's memories.

What about them? Do you think the emergency conference call with Hostile Nation Airlines begins with everyone reading off their preferred pronouns?

If someone's wife is such a miserable person she can't help you out in an emergency, she is an awful person and you made a mistake marrying her.

Do you think an airplane mechanic is married to an arrogant, self-important executive who is far too busy for packing bags to help you out in a bind?

If you're a man with a wife, and you need your partner to bring you a suitcase, then you ask your wife to bring you a suitcase. There's nothing to read into here...

Agreed. And knowing that this is a regular occurrence, one should have a packed travel bag with them at all times (aka bug-out bag, go-bag, shtf-bag).

That’s also a hassle. There’s no reason to claim there’s a right way to handle this sort of situation without knowing *significantly* more about the specifics, which we don’t.

-SOP when I did this kind of thing in the offshore business was to always have a bag with your kit and a passport in the boot of your car.

The first few times I called my wife to tell her I'd be late for dinner as I was on my way to Macaé or Penang she thought it was amusing. Her enthusiasm soon waned, and luckily I was bright enough to ease up on the unscheduled traveling before she really tired of it (and me!)

From the photo at the bottom, the airline is Air Seychelles. And from that we can figure out it was in Paris: https://www.nation.sc/archive/218414/air-seychelles-b767-300...

The sleuthing I was hoping to find!

Ouch, ground crew manouvered the plane into the collision, must've been an expensive mistake for the airport.

And now it's flying cargo for Amazon.

https://www.airfleets.net/ficheapp/plane-b767-26328.htm

They papered over most of the identifying registry numbers except one, under the wing. And you can see it here.

https://www.youtube.com/watch?v=ZGjaT-0x7z8&t=4m01s

Here's an AP story about the 747 that flopped into a muddy field when a takeoff was aborted:

https://timesmachine.nytimes.com/timesmachine/1988/07/25/491...

Runway Accident in India NEW DELHI, India (AP) — An Air France Boeing 747 jet carrying 275 peo- ple swerved off a runway at New Del- hi's airport and crashed into a field to- day after the pilot aborted takeoff be- cause he thought an engine was on fire, airport officials said. No one was seri- ously injured in the accident, the Air- port Authority said. Indian news agen- cies said several passengers on the flight, which had been bound for Paris, were treated for minor cuts and bruises after evacuating the plane through emergency chutes.

How do they work around visa requirements for visiting engineers? Wouldn't that lead to few days worth of delay at least? They mention New Delhi in the article - Americans require visas to visit India.

They also mention filled passports in the article. One assumes there's a team at Boeing whose job it is to ensure each increment comes as smoothly and quickly as possible.

If it's a national carrier or a major airline, I think they can probably contact people high enough in the government that visa issues will be quickly resolved. If Air India is losing 200k a day because of a plane being inoperable, I suspect that they can get the Indian government to expedite any visas and allow people to legally stay long enough to fix the plane. Anyone on Boeing's team is probably really low risk in terms of overstaying their visa illegally or committing some heinous crime.

Heck, airlines can do that just for dumb passengers.

I was (attempting to) check in for my NZ to Australia flight on Emirates, and had totally forgotten Americans required a visa to enter Australia. They called ahead and got me a visa on arrival - all I had was a piece of scrap paper with a number written on it that I redeemed at the other end.

It worked out for you because USA - Australia visa requirements are pretty lax. Generally speaking, developed countries have relaxed (or no) visa requirements for each other. Had you been traveling to say China, India, or basically any global south country, it wouldn't have worked out.

Yep, definitely American privilege making up for American bad assumptions. I have reliably gotten my visa requirements right the rest of the time across said global South.

Most countries can fast track a visa if there is enough money on the line for someone.

Before I had kids, I did pretty much the same thing for a supplier of offshore handling equipment - seismic survey, oceanography, ROVs, that kind of thing. On more than one occasion we were sent out with tickets only to some major hub in the general vicinity of where we were going, giving the back office a few more hours to figure out how to have a visa or a waiver issued by the time we got to the hub.

Whenever possible, we had work visas issued in advance for most countries we were likely to visit - oh, and a pair of passports to ensure, uh, incompatible visas didn't end up in the same passport.

(An Iranian immigration officer once chucklingly suggested that if he could offer me just one bit of professional advice, it would be to NOT go to the US on the passport currently on his desk - with visas to Iran, Libya and Algerie, to name but a few...)

I was surprised to learn that the US government allows duplicate passports for exactly this kind of situation. https://travel.state.gov/content/travel/en/passports/have-pa...

If they stay airside at the airport they don’t go through immigration. Just shack up in the hangar.

I don’t think that’s an option generally speaking. You’d still need to go through security to get the the authorized worker area to be checked again and passes issued. It’s not like you can just bust open a door and jump out to the tarmac without some sort of pass and ID

The initial assessment team is probably flown in as crew. Those are exempt from visa requirements.

The full repair team with equipment and parts definitely needs to get through customs (but there are import duty exceptions for airplane spare parts in international treaties), and get visa arranged etc.

Every airplane is unique. And considering comparatively low tech planes from 20+ years ago required 2-3 dozen (physical) maintenance manuals, one can only imagine the number of virtual pages of documentation required to know the details and repair procedures for each airplane today.

For every series and subset of aircraft there may be countless special maintenance directives that are issued to remedy risks and problems. Managing those document updates is a big deal, and knowing exact and correct details of each plane would be essential to rapid repair (and knowing exactly which parts to bring).

Not to take anything away from the amazing traveling repair experts, but none of this would be possible without HN-type folks building the data management and retrieval systems necessary to support it. So yay for us too.

> … one can only imagine the number of virtual pages of documentation required to know the details and repair procedures for each airplane today.

As much someone can write a procedure manual about how to fix something and teach it others, there is still a level of intangible skills, which come with long-term experience, to fix something that broken in an “organic” way (i.e. crumpled aluminum) that can’t be easily described in words.

This is what’s “magic” about these types of rapid response teams.

Maybe said another way: there are textbook failures and then there are real world failures. Usually the real world failures are problems that go far beyond the immediate technical failure and stretch into incentive structures, human fallibility, deeper physics, and unluckiness. Those failures are worth sharing and you won’t get help solving them from ChatGPT.

Boeing tracks every part in every airplane.

There's an (in)famous legend that they use an Excel spreadsheet for this, or did at one time.

I don't buy that. I was at Boeing 10 years before Excel, and they tracked every part. I don't know how much of that was computerized.

Besides, can you really track millions of parts on a spreadsheet on a computer with 640K?

Yeah clearly at some point in the past spreadsheets didn't exist but Boeing was building airplanes.

I went googling for the page where I had first read this rumor, and could not find it. I did find this, which I had never heard of.

https://en.wikipedia.org/wiki/Boeing_Calc

The reason for the tracking is if a bad part turns up, Boeing can find every part that came from that batch and inspect/replace it.

They may currently be busy with working on DHL's 767 that has been grounded in Beirut aber beint "bent" at a hard landing. But I am not sure if that is really fixable:

https://nitter.net/aviationbrk/status/1707102471913349398#m

There was a United 767 that had the same thing back in July. It's been scrapped.

https://www.airfleets.net/ficheapp/plane-b767-25091.htm

On August 12, 1985 over 500 passengers on JAL 123 died due to a faulty repair by this Boeing team.

A faulty repair on the exact same component that's the centerpiece of the article, no less: the rear pressure bulkhead. Maybe JAL 123 is why they tend to replace them instead of patching them...

Seems like this can turn into a "customer breathing down your neck" situation. The incentives and stress probably don't mix well with a safety first attitude. Imagine going to your customers business, into the belly of the beast and they're livid that your product is broken and it's costing them $200k a day. Your job is to fix it asap. All the rules and procedures in the world won't prevent your team from feeling rushed.

Well yeah. From the article:

> In 1978, a Boeing AOG team repaired the bulkhead of a Japan Airlines 747 damaged in a tail-drag incident. Seven years later, the repair failed in flight, resulting in an explosive depressurization that tore off the vertical fin and severed all hydraulics systems. Some 30 minutes later, the aircraft slammed into a mountainside; 520 people died in the second worst airline disaster in history. Investigators determined that the AOG repair did not comply with Boeing’s own Structural Repair Manual. Boeing accepted 80 percent of the liability for the crash, while JAL accepted the remainder for neglecting signs that the repaired bulkhead was weakening.

https://en.wikipedia.org/wiki/Japan_Air_Lines_Flight_123

To be fair, that incident is extremely isolated. This team repairs on the order of 100 airplanes a year. They had one major failure in the past 50 years? I am certainly not excusing that nor the loss of lives, however I can not think of any other team working on non-trivial problems, where each problem is unique, that has a lower failure rate.

And it’s weird the repaired plane wasn’t given a very thorough inspection the next chance it could.

> They had one major failure in the past 50 years?

Citation needed...

The incident was citied in the fine article. And it is a well-known incident - other than Tenerif and the 9/11 incidents I can not offhand think of any other incidents with as many fatalities. The Airplane On Ground team (I'm pretty sure that's what AOG stands for, maybe it's Airliner on Ground) is more-or-less well-known in aviation communities.

> All the rules and procedures in the world won't prevent your team from feeling rushed.

While feelings are personal, team willingess to take unapproved shortcuts is a management and regulator item.

Reply on news.ycombinator.com