Hacker News Reader

Comment by freedomben | Hacker News Reader

In the mid 2000s I was in the Middle East and used the internet to connect back to the United States for conducting business. One day almost every site became unreachable (anything hosted in the states). It turned out that a naval vessel had dropped anchor and Murphy's Law put the anchor right on top of some undersea fiber cables. It was called a freak accident because statistically was so unlikely to happen.

It took a while to figure out what happened, and internet was out for a couple of days. Considering service was restored in 30 minutes in this case, we are getting a lot better at handling severed cables.

View on HN

> Considering service was restored in 30 minutes in this case, we are getting a lot better at handling severed cables.

30 minutes?

This is absolutely mind-boggling to me. 30 minutes -- and you can identify exactly where the cut was, deep in the water, access it, fix it in the water? To me that sounds like a 30-month job not a 30-second job.

Anyone here with insight into this fix-process: could you please shed light on how this is done and how it can be so fast?

I'd guess traffic was rerouted.

I'd guess it was done on purpose, so that it be rerouted.

My first guess would be re-routing traffic away from the damaged cable(s)

It’s about a 1-3 month repair job to acquire replacement cable, load it into a ship, and sail out to the break point.

However, service is restored sooner than that by rerouting service over a different fiber cable or undamaged set of fibers.

They don't replace the cable typically, they fuse it with epoxy and it usually works pretty efficiently, mostly taking the time scheduling the boat and equipment the repair is likely a couple hour job at most and 12-24 for the epoxy to dry.

The cable is not fully replaced. They lift the cable up to a boat and repair it (replacing the damaged bit and joining it to the original) and lower it back down.

Here is an animation describing the process https://m.youtube.com/watch?v=ZJblHOGXKo4

Amazing really that the optics at the repair site are still good enough to carry the bandwidth.

No one fixed the cables, they have rerouted traffic through other links. Most of the traffic was automatically rerouted, that's how BGP works.

If it took 30 minutes that's not BGP, that's some people at ISPs manually provisioning new circuits on DWDM systems, using a moderate amount of automation tools, and bringing the circuits up facing routers at both ends. Or doing something like creating a few new VLANs in existing larger 100/200/400Gbps intercity transport links, and trunking those VLANs through to routers on each end. A topology change in BGP is a couple of seconds.

That's why I've specifically wrote "most", of course it's not all BGP, BGP routing algorithms do not need 30 minutes to change routes.

Do international peers tend to use BFD?

BGP may or may not still be involved.

BGP itself has no real facility for dealing with circuit congestion.

So, I am willing to bet that dynamic routing was certainly in place, but the secondary/eBGP multipath destinations were full when the primary path went down.

And that necessitates manual action in instances where you aren't doing MPLS-TE, or don't have enough standby capacity, or both.

Keeping sufficient hot standby submarine capacity live and in contract is very costly in $/month.

No, it's a process of ISPs rerouting transport traffic. Onto other routes at OSI layer 1.

Physical repairs of broken submarine fiber takes weeks to a month.

An event like this, where multiple fibers are disrupted and magically fixed less than an hour later just smells like the NSA, one of the other five eyes or a friendly telco installed a fiber optic beam splitter.

https://en.wikipedia.org/wiki/Beam_splitter

I don't have any traceroutes so this is speculation but the probably just route around it if it only takes 30 minutes. I was under the impression that a beam splitter can be installed without disruption. If they fucked up and broke the cable installing a beam splitter, a 30 minute fix is still pretty impressive.

I‘m by no means an expert but I did some datacenter work. I wonder how this would be possible to install a beam splitter without disruption? Don’t you need to plug the fibers out, put the splitter in and then plug the fibers back in the splitter? I was searching YouTube but couldn’t find a video.

You don't need a beam splitter to listen in on traffic.

They bend the cables, thus allowing part of it be tracked with some residue light coming out of the fibre.

I agree, this might be possible but I wonder how good the quality is then? A splitter will give you 50% light. That limits the length of the cable too, doesn’t it? How many errors will be there when you bend the cable and get maybe 15% light of encrypted traffic, and you can’t signal any retransmission?

On run of the mill singlemode fiber loss is roughly 0.5dB/km. Splitting off half of the light (and you don't need half, much less is fine), would result in 3dB of loss.

Fiber tramsreceivers automatically negotiate the power needed to sustain the link. So in case of datacenter or city networker, you could do a split-of without introducing too much loss and error rates.

Trans-oceanic fibers are different beast but I guess someone figured it out.

The light is kept inside the fiber by the change in refraction index in the radial direction of the fiber so there is a total internal reflection. But if you bend the fiber enough then reflection is not total and part of the beam goes outside.

Rerouted to a backup, I'd guess?

Identifying the break can be done really quick:

https://en.wikipedia.org/wiki/Optical_time-domain_reflectome...

But then you still have to go there and fix it. I can't even get a pizza delivered in 30 minutes so I suspect the fix did not actually repair any cables.

I understand your astonishment, but if they had properly configured dynamic routing, it would not have taken that long.

Sadly some international ISP’s don’t like to share (even sanitized) routing tables...

LOL... I'm so sure the US boats seem to keep 'accidentally' being around when fiber is cut. You do know US/NSA actively taps global fiber networks. I assume they cut it in one place "accidentally" while they tap it in another to avoid being detected.

Why only mention the US here as if they're the only ones that do it?

It's routine activity for superpowers to tap undersea cables.

"superpowers"?

The countries with the economic and military power to do so. Today, likely the US, Russia, China, and the UK on the periphery.

US is the biggest and done it the longest. It's easy to act like the US is thee super power. But when you look at it, no other 'superpower' holds a candle to our 800 foreign military bases.

Uh... you don't need to cut fiber to tap it, you have to just bend it to allow the light to leak.

Performing any worthwhile tap that way would show a measurable difference to anyone monitoring it. When you cut the line and connect at the same time you would be resetting the baseline to be measured against when someone starts monitoring again.

Taps are almost certainly being done by submarines -- there are even books about it[1].

Why would surface ships be around, blatantly giving away what is happening? The likeliest answer is that they screwed up with their anchoring procedures.

[1] The best known being Blind Man's Bluff. See also: https://en.wikipedia.org/wiki/Operation_Ivy_Bells

My guess would be they create an excuse while elsewhere do it. I don’t picture a boat on top of a submarine performing the procedure. The boat{excuse} is the excuse that renders any monitoring of the signal useless when the tap is done.

Anchors cutting internet cables happen all the time. It is a weekly occurrence. If you want a visual image: imagine a plane laying down a cable over a forest. It is an heavy cable designed to crash a bit inside the canopy, but it is likely to still dangle in many places.

Another super powerful plane now lays down a one ton hook on a big ass chain and drags it around the forest until it settles on something strong enough to stop it. It does not make the coincidence that big, especially when you consider that many cables land close to commercial routes.

Wait, this is actually how internet cables are laid?

Mostly yes, they do protective measures like burying in highest risk waters.

(The surveying involved in route planning prevents dangling the cable in nonworkable ways)

Well no, they use boats, not plane. ;-)

But mostly yes. It is interesting to read a bit on submarine cables that our infrastructure depends that much on.

It is for instance interesting to know that their landing points are considered strategical locations and that at least US and Russian navies have trained at eavesdropping on them using submarines. Just throwing that in case you are wondering why you don't hear much about X-37's ability to rendez-vous and "repair" satellites.

Wasn't there a woman in Armenia or something who cut a cable in her house the internet for the whole country dropped. I find it difficult to believe what I just wrote but I recall it.

Ah: https://www.theguardian.com/world/2011/apr/06/georgian-woman...

She was in Georgia.

Hummm from reading the article it doesn’t seem like she just “cut a cable in her house” but rather vandalized a major optic fiber cable while trying to steal copper not on her property.

is she supposed to mine bitcoin or infect pages with hivecoin? just leave her be

I'm definitely leaving her be! Do you think she's going to come and read hacker news comment and be offended? We don't even know who she is. I'm just correcting facts here.

Off the top of my head I could believe that a good place to lay cable (by some definition) might be a good place to drop an anchor. The more shallow the water, the less cable you need.

Indeed, and so many ports are attached to major cities that are also great places for internet hubs ...

Cable landing stations are generally a bit further away from ports and such. They're also heavily armored when near the shore for this particular reason. The folks laying the cables will also put in energy to do outreach with local fishing organizations to let them know about the cables too (and they're charted).

On 2006-12-26 there was an underwater earthquake in Taiwan that cut off all sea cables between East Asia and Europe.

I was in China at the time. All internet traffic was going the other way around the globe and was extremely slow or most often just timed out.

It took weeks to get normal internet speeds again.

Underwater volcanoes (rock slides) are the biggest threat to cables when you get into the deep ocean. The cables are much more thin (less armoring then when near the shore). The other fault encountered in the deep (no anchor risk) is from amplifiers failing on the cable which is a long process to get a ship out to replace.

what are you talking about here, guitar amplifiers? some radar equipment falling from a ship? optical signal amplifiers?

EDFA's: erbium doped fiber amplifier. They're optical signal amplifiers. Due to the attenuation of the fiber, they need to be installed every so often on the fiber, under the sea. Since the equipment is cheap in comparison to the cost of repair, modern undersea amplifier boxes often contain multiple amplifiers and switchgear to MUX to a new one after it fails.

This. If you see the oceanic cables every x amount of kilometers you'll see a bump in them: that's the amplifier. There are also junction points too where cables can split off to serve locations on the route. Newer cables have the ability to dynamically reconfigure these (wavelength switching) these junctions rather than be static (the older way).

very interesting, thanks both. I'm surprised something travelling at the speed of light over a relatively short distance attenuates at all. I wonder if these amplifiers are purely optical (like mini magnifying glasses), or if they are powered (requiring running adjacent power cables)

Wikipedia has a lot of details about how the amplifiers work: https://en.wikipedia.org/wiki/Submarine_communications_cable...

It’s traveling through glass and the glass has imperfections. Can you imagine trying to see through glass 30 miles thick?

Power is involved too. When I first visited a cable landing station I got to learn about the process to de-energize / energize the cable and coordinate with the ship and the remote side. Pretty involved but really neat to see. The stories are interesting too. One cable landing station on a small island the main tech would bring his family in during a typhoon to shelter there for days since it was a fairly protected site that had plenty of fuel for the generators.

More like there was a cable cut, resulted in congestion on someone's network and they rerouted it (internal backbone changes, etc.)

Just another case of, influencing without authority.

Reply on news.ycombinator.com