Hacker News Reader

> So maybe there is a plan, or maybe they just designed the system in such a way that they could convince themselves that it would not go down unless things were so severe that loss of the phone system would not be your chief worry.

My belief from working in very large companies, and (previously) in mission critical systems is that a clean bootstrap and recovery process is extremely unlikely, almost impossible. Because in complex systems full of legacy parts and people who have long retired, the stars won't align.

The only way to truly know is to design and periodically test for disaster scenarios (emphasis on the plural). But due to the scale in time and space, cost and bureaucracy, this planning and rehearsing is not going to happen with the desired detail and intensity. People do not seriously plan for things that have never happened.

If it does happen, there will be a small group of extremely capable people that will find a way to bootstrap the system. It won't be according to some previously laid out plans -- they will make the plan in real time. They're not famous and probably never will be.

Eh, would not surprise me in the slightest if there was a secret billion dollar program that specifically practiced disaster recovery for the phone network. The government doesn't have the same motivations as a business and they spend a lot of money in a lot of places just to be prepared for unlikely events. Like we spend billions on military hardware we don't forsee ever needing _to keep the engineering capacity to design and build military hardware_.

>Like we spend billions on military hardware we don't forsee ever needing _to keep the engineering capacity to design and build military hardware_.

Well, more because the military industrial complex lines the pockets of your politicians, who in turn decide how to spend the budget.

The two are not mutually exclusive. In fact they reinforce each other.

In Ireland, we just build a runway to allow the plane to take off again.

https://www.rte.ie/archives/2018/0521/965058-mexican-lands-p...

Clear evidence that news was more entertaining in the 80s:

    In the merry month of May
    Just before the dawn of day
    A plane flew in for Shannon to refuel.
    Because Shannon is fogged out,
    Their are the rite of ought
    To touch down in Cork Airport as a rule.
    
    As he flew towards Mallow town 
    His supply of fuel was down
    But the pilot was as cool as cool could be.
    In a racetrack west of town
    He made a safe touch down
    Just beside the Mallow sugar factory.

Hunt the hare and turn her down the rocky road... Wait, what?

this is truly one of the takeoffs of all time

And in the meantime they had the pilot judge a beauty contest. What a story!

I've long had a fascination with aviation incidents (the Gimli Glider happened on my birthday), but I hadn't heard of this one before!

What a great story. Thank you for posting it.

Gimli Glider incident: https://simpleflying.com/gimli-glider/

It appears the town is still celebrating the event: https://www.independent.ie/regionals/cork/news/family-of-mex... Publication date is 2023-03-22, so 40th anniversary.

There's the story of TACA 110, a Boeing 737-300 which landed on a grass levee, and took off from an auto-traffic boulevard (albeit one built on a prior NASA airfield runway).

<https://en.wikipedia.org/wiki/TACA_Flight_110>

If you dig deep enough into the SS7 stuff running in a modern regional ILEC it's way more fragile than you might think. Mostly because it's no longer treated as an absolutely cannot fail thing that is also a primary source of revenue like back in the days when everyone has a POTS line and tons of money came in from long distance bills. Many operators are decommissioning stuff like 5ESS and Nortel switches and moving to modern soft switches as quickly as they can.

The network stuff underpinning a lot of critical tdm phone traffic these days is like a collection of 23 year old Cisco 15454 held together by spare parts and a few people who care about them.

I've been out of the industry just long enough to remember that Cerent 454's got rebranded as Cisco 15454's right near the end of my tenure...

Yow. Way to make a guy feel old! :P

Mostly I was using the 15454 as an example, there's lots of other 20 to 30 year old stuff out there in the TDM transport sector that's only available on ebay, through weird used equipment dealers, or by finding a decom from another ISP/telco.. Stuff like T1 (or DS0!) to DS3 mux/demux to attach to a 15454, or similar. There's literally 911 call center transport circuits being held together by the telco equivalent of duct tape and string right now, nobody notices until it breaks.

One of the weird challenges in building a new state-of-the-art inter city DWDM transport network now is dealing with things like legacy customers that have one OC48 and are unlikely to drop it any time soon, it's a considerable monthly revenue source, and have to deal with stuffing that into the system along with 100Gbps and greater coherent circuits.

Also from a customer relations perspective sometimes the customer literally forgets that they have this extremely exensive DS3 or OC48 or something in monthly recurring billing, and you don't want to bring it to the attention of management, because they might go "are we still using this?" and cancel it.

> There's literally 911 call center transport circuits being held together by the telco equivalent of duct tape and string right now, nobody notices until it breaks.

And break it does.

https://www.kron4.com/news/bay-area/911-dispatch-system-in-o...

I contracted for a telco who held a national emergency number contract, and after they fired all the old engineers and the inevitable outages started happening, had to budget a significant amount per month for flowers for families of the deceased who weren’t able to reach anybody during heart attacks etc.

It would have been better if it was really big SLA breach fines however.

The 2001 shutdown is even more crazy when you consider that for the FAA administrator who ordered it, it was his first day on the job. Hell of a first day.

Hell of a day to quit sniffing glue.

> I don't believe I heard that any planes landed where they could never leave.

This has happened before outside of 2001, albeit not really a DR issue but more a political issue -- if we look to the Meigs Field destruction by former Chicago mayor Richard Daley, multiple aircraft were left stranded with a now destroyed runway. (The solution was to just give them special clearance to take off on a taxiway, but still.)

https://web.archive.org/web/20110720045652/http://www.aopa.o...

OP here. Thanks for the remarks! That tale is apocryphal to me. I found it amusing, and telling in the sense that disasters are the last thing people think of, and for large enough systems (especially ones that have accreted over human generations) the organisational knowledge has probably been lost to retirement and death. And if you're lucky, maybe a scrap of it has survived. Then one's job is to do the requisite software archaeology to figure out what one's people might have entirely forgotten.

Also, if that script remained valid at the time, I doubt it would do any critical actions. It might have been a sort of literal script to follow --- run the script, see what it says, do a thing, run the script, and so forth. Its supposed job was to help humans solve a bootstrap problem.

I see how my wording of that passage makes it sound like the be-all end-all of cold booting a telco. But that's what we get when we wall-of-text in our Slacks :D

(edit: clarifying remarks)

You know you can actually bring fuel to a plane and refuel it wherever it is? And that’s probably the answer to the first question, it can be done but you you have to have someone who knows what he’s doing think outside the box a little.