Pretty grim that a life critical system wasn't designed to report that the backup fibre was unserviceable until they attempted to switch over to it.
I wonder how long it was down? Days, weeks, months?
I am paying, $1000, $1800 & $1900 for the same service at 3 different location (20 mins from each other).
Two locations, I also have old coax lines that are still active, but not paying for it.
When I bought two businesses, I learned that they were paying for a dedicated fiber but using coax service.
At one of the location, we had fiber, and paying for backup coax and wireless. But if you turn off fiber box, it wont fail over to either one.
I wouldn't surprise it was down for weeks and no one bothered about it.
I thought about what I believe is your way until https://www.tomshardware.com/opinion/t-mobile-home-internet-... and https://www.tomshardware.com/news/t-mobile-misleads-home-int... .
Not my area of expertise though so I could be leaving out info
Edit: Starry is the company, I believe they were just purchased by Verizon
Had a few issues in some lamdlpcked counties on occasion - Kabul had all traffic routing via the same route into Pakistan for a few years, Kathmandu had an earthquake knock out both ISPs within 30 seconds of each other.
Backups via starlink global roaming work well now in particularly challenging environments.
Luckily we were on Century Link so weren't affected by his stupidity. lol
Over ten years ago, my employer was spinning up a new DC across the state and had three links between it and the primary DC. Two were pretty direct, but we needed a third because at one point in the 300 mile path, the two main links went within 400 meters of each other.
So how national-security-adjacent critical systems like an airport system doesn't have a larger set of backup links, and validates that they are geographically separate up until the connections, and have realtime alerting on the connection status, is surprising to me.
This isn't relevant in this case. The problems occurred at different places. The backup fiber was, separately from the issues with the primary system, cut by a construction crew. It's not two fiber lines both cut at the same spot.
You can get two separate failures in short succession, where the first one hasn’t been fixed. For critical devices that’s why you have tertiary backups, ideally on different technology compel you. For one major event I had diverse fibre and satelite and microwave. Lost microwave and satelite to electronic warfare but the fibres were ok. Ofcom were really pissed about it, but nothing they could do.
No?
Like, okay, monitoring reports what there is no link.
Now what? Somebody gets a hi vis vest, a hard hat and goes along the cable to investigate the reason?
There is nothing monitoring could help, especially if the cable was cut recent enough.
> An RFTS enables fiber to be automatically tested from a central location. A central computer is used to control the operation of OTDR-like test components located at key points in the fiber network. The test components scan the fiber to locate problems. If a problem is found, its location is noted and the appropriate personnel are notified to begin the repair process.
If it's down for more than 15 minutes in country or 2 hours internationally it gets flagged for manual attention. Shorter outages are logged but only looked at monthly as part of the operational report process.
Now sure you can have two independent faults on the same day, but the reports are that they didn't know the backup line was down.
In your case you already have way more than monitoring. You have the infrastructure designed for resilience. You have that design implemented. You have the protocols to do if the things go north. You have automation to disregard minor events and to bring to the attention more serious things. You have way more than monitoring alone.
For a major trans-oceanic backbone provider, at least 15ish years ago they had a mile or two between Detroit and Chicago where both ends were on the same side of the interstate highway.
But it's more frequent on DAS (Distributed Antenna Systems, AKA small-cell or micro-cell) networks.
Also the challenge of when fibers are leased (if that's still a thing, based on the networks I helped design I'd say 'probably').
They really are analogous to Lamport's "Distributed System" quip; A damaged fiber owned by a company you have never heard of can wreck your day.
Fibres on the Texas link die all the time (yearly) but that’s fine.
My circuits in the far east are often carried one via dues and one via the USA.
It is possible to guarentee diversity, you just need to get specific detail and have good contractual protections.
Yeah, FWIW the non-redundant DAS links we designed when I was in that industry, it was typically a case of the client telling us 'That is too expensive, we accept the risk'.
The Detroit-chicago thing, I don't know the story on that, it was something that fiber provider had already built.
For mission critical stuff like airports I would like to think they're go for a more rigorous methodology than hope for the best on paths
Usually it’s a combo of a tier 1 that decides that a resell agreement is “good enough” and “well just eat the loss” and then by the time stuff like this rolls around “we’ll get back to you” + a bunch of silly explanations that boil down to “you’re not gonna sue us though” start coming out lol
Then there's another faction that doesn't trust a word suppliers say, who will insist on two different suppliers, as while in theory they could be sharing the same routes, it's far less likely.
Comes down to "do you trust contracts".
Of course this is for third tier sites like branch offices. Major sites require far more oversight, including visibility of cable routes and any hops on the internal network
You're going to be hard-pressed to find any point in the American empire's life when it doesn't have 'current tensions' with someone or other.
Similar events happened in Europe disguised as thieves stealing fiber optic. This does not have any sense economically, as the value in the market is zero so... either is an honest accident and is cleared in a few days, or is sabotage
Do you honestly think that crackheads think that far in advance?
I've seen fiberoptic cables stolen from 2 (city) jobsites in the last 5 years, once by tweakers later caught trying to sell them as scrap copper and the second thief was never caught. This happened even though the spools had big signs on them saying "Fiber Optic Cable - NO COPPER".
Crackheads can be manipulated easily, or bribed. Maybe the value was in that they were paid with $50 in drugs for doing that.
People too dumb to plan X and to inconsistent to spend several days honing the plan, somehow avoided all the traps and locks, go for the precise location and do X... either they worked in the place and are very familiar with each room, or there is a director manipulating them.
With the added value that addicts can be overdosed easily if they try to strike back and treat to getting too talkative. Nobody would bat an eye.
Recently, I tried calling 811 before digging in my yard. The webpage was broken and the hotline kept me on hold forever. I gave up. Small wonder.
https://blackhydrovac.com/underground-utility-strikes-learn-...
Plenty of people don’t bother and YOLO it, and usually it’s fine - until it isn’t.
The punchline is that it's about 2000 ft from the local CO, where (I believe) half the town's lines terminate.
A lot of people are incompetent.
I worked at a company worth a few billion and the leadership balked when they told engineering they wanted a near instantaneous failover system and our department informed them that would require paying for a second environment that could be rolled over to.
It is rare to find leaders who can accept the cost of redundant infrastructure that is there for emergency backup.
What puzzles me is why they can’t accept it when they are perfectly fine with insurance costs and I can’t see much of a difference between the two when looking at a spreadsheet of costs other than possibly tax differences between the type of expenditure.
Redundancy has costs and those costs can be spent elsewhere like having higher quality or bigger disks, or a faster network switch or better CPUs etc.
In theory, it would also be better to own two cars instead of one, because what if the first one gets in an accident or just breaks down. Yet, not everyone can afford that. Should you just buy two half-as-expensive cars than what you can buy one of, so you can say you have a "backup"? Likely the two half-price ones would be so much crappier that the one good car would cause you less trouble in expectation than driving a shitty one and then having another shitty spare one, both of which will constantly have issues.
And in other labs I've already seen data loss because they messed with their setup in incompetent ways.
You can't always have everything. There is a budget. There are storage needs. You can try to reduce your storage and instead beef up the backups and stuff, but then you can barely do anything and people have to constantly delete their data, and worry about space and that slows things down, generates hostility and conflict between colleagues, and just causes pain in general. You can't see these things from just a technical desirability point of view or what setup would get the most upvotes on r/DataHoarder or whatnot.
There's generally two wrong responses: (1) We spent a lot of money on 'blah blah blah', a lot of other companies use it, so yeah, we've got a backup/failure system. And, (2) inadequate testing - either, we tested 1 of 50 services, and it worked, so the whole system can be restored; or, we gracefully tested, and it worked, so it will obviously work during not-graceful incidents.
And the root cause of this is generally that no one gets promoted for implementing an adequate backup/failure system, or it's extremely rare.
Not saying that's what is going on here, just that it's possible.
I worked for a regional ISP that had a major outage when the redundant fiber provided by the telephone company was cut in one place triggering a full loss of connectivity. It also caused a massive 911 outage for 200,000 people as it isolated the 911 center from the city core.
Turns out the phone company didn't connect one side of the ring topology even though they certified they did. Needless to say lawsuits abounded.
The amount of shit I was given everytime I went through a checklist working at a fedramp certified company working with emergency alerts on the phone system made me prematurely gray.
I could feel the barely contained seething rage everytime I told the execs that the reason this release will take 3 days and not be instantaneous like your friends releases at a faang and that information made you embarrassed at your dinner party, is because you agreed to this process contractually years ago and now it’s a crime if I just sign off on it being ok without actually checking that it’s ok.