All servers at OVH North American datacenters offline due to fiber cut | Hacker News Reader

All servers at OVH North American datacenters offline due to fiber cut

status.ovh.net

85 pointsck243 comments

Constructing a brand new dark fiber network over long distance requires an insane amount of work, plus a ton of legal and administrative issues to deal with.

OVH had started working on the construction of a second dark fiber network a long time ago, with at least the capacity of the previous one. Due to a strange quirk of fate, they had planned to complete the construction within the next few days.

After the fiber cut, they not only tried to locate and repair the cut pairs of cable, but also simultaneously rushed to finish the construction of the new network. They made in hours what was initially planned to require at least another week.

The services are now back online at full speed, because they managed to finish installing the new cables, test them and connect the whole data center right away to the new network. And it worked.

That's a story to tell your kids and grandkids.

I have such great respect for network admins. I have no idea what it takes to get a network to route properly worldwide on multiple trunks.

Long time OVH customer here, they had a similar issue on May 2015 for the same DC in Canada where a car hit one of the poles along the BHS/MTL via the North route, damaging the optical fiber cable: 2 of the 3 pairs of cables where down.

http://status.ovh.co.uk/?do=details&id=9603

At the time, they told customers they were adding new fiber providers and cables.

Laying new fiber takes years. They turned it on today, so it must have been in the works then too.

This is why you have fiber leaving your datacenter in two different direction and to two backbone providers. Once you do your due diligence, it's really hard to guarantee your providers never share infrastructure. Good luck guys!

Also, be aware that fiber sabotage is a thing that happens. It's rare, but it's bad, and in my experience even top-notch providers take days to recover from it.

Even if you aren't the target, it doesn't matter how many redundant links your single uplink provider has if the threat model is hostile actors with their infrastructure mapped out vs. wayward backhoes.

Truth - stories of redundant paths getting groomed into the same cable years after provisioning seem to be all too common.

The effect on the weathermap is "clear cut": http://weathermap.ovh.net/#usa

It looks like they still have one link running, but a small saturated one.

Yes, technically the DC is not down... I can still ping forum.ovh.us and the webpage loads, but very slowly (as excpected).

800gbps of connectivity cut! yikes, that would put all hands on deck... I'm hearing issues with Verizon/Terremark and Wellpoint networks from clients, wonder if it has anything to do with... godspeed OVH

Multihome your edge, folks.

Glad to see this instead of bashing on OVH. This can happen anywhere. Every data center is at the end of one or maybe a few very thin little cables that can be cut very easily.

Fiber cuts are definitely not OVH's fault.

However lack of additional routing might be.

Apparently they were right in the middle of installing additional fiber on other routes for the datacenters but they are still dark and not online yet. They claim they are trying to accelerate getting those links online in their french ticket

http://travaux.ovh.net/?do=details&id=15238

Reminds me of the joke: Traveller to ops guy, "Why do you have a piece of fiber optic cable in your rescue kit?", ops guy, "That way if I ever get stranded I just bury this cable nearby and wait for a back hoe to show up and cut it!"

On a weird note, when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber. Sure its great that you have two providers but it is only one fiber didn't seem to sink in. Pass.

The problem is very real of course, when a disgruntled AT&T engineer cut lines in South San Jose they did a huge amount of damage. Looking at their impact and the time it takes to resplice a fiber, a group of us calculated it would take no more than a 100 people to take California completely off the cabled Internet (satellite is really hard to disable from the ground). That seems pretty risky.

There are actually reports of people being spotted testing major connection points for fiber and power at various places (I think in California).

Not sure if it is real or trumped up, but if real, it would be domestic terrorism for sure.

Maybe this what you're referring to: http://www.wsj.com/articles/SB100014240527023048511045793591...

https://en.wikipedia.org/wiki/Metcalf_sniper_attack ?

>when evaluating data centers I had one tell me they had redundant connections to the Internet but what they really had were two lambdas on the same fiber.

Sadly, this kind of shenanigan is pretty common.

I've also seen datacenters that have additional providers on different fiber...but when you follow the fibers upstream, all fiber winds up at the same POP. Predictably, outages at the POP would take the downstream datacenters offline.

One of the biggest backbones in Finland had both a major fiber and its backup running in the same ditch. Surprisingly the same excavator broke them both at the same time.

It was a great opportunity to learn who your local ISP buys international transit from.

Normally, I'd agree.

However...OVH hasn't installed sufficient redundancy to safeguard against this stuff for years and its happened on an annual basis.

http://status.ovh.net/?do=details&id=6629 http://status.ovh.net/?do=details&id=4376

Except what they completed today is the construction of another dark fiber network to this data center, going through a completely different physical path.

After 5+ years? Finally?

I'll believe it when it doesn't happen in 2016.

This will certainly mean a temporary drop on botnet activity!!!

For customers of dedicated servers, remember to open a support ticket so you can get compensated accordingly. (default SLA = 5% per hour of downtime)

https://twitter.com/rpurinton/status/661255883582775298

Now lets correlate this with "bad" TOR sites downtimes to narrow down their origins...

The OVH is great. A really good solution for many use cases. Of course you must know exactly what they are. Isn't a good idea trust a single DC setup, even worst if they are totally dirt-cheap. Unfortunately people misuse them.

Up!

http://status.ovh.co.uk/?do=details&id=9603

Looks irrelevant, all comments seem to be from May.

Notice the date of last update on that status: Thursday, 23 July 2015, 11:46AM

You're looking at the wrong one. The downtime today is this one:

http://status.ovh.co.uk/?do=details&id=11304

And as of right now (4:17 PM CST), it says it's still being worked on:

> We still don't have an ETA regarding the repairs on the existing fiber.

> However, it's been confirmed that 2 out the 3 new pairs of the Eastern route (which were being setup) are now usable. We are working with our providers in order to get a full connection as quickly as possible: the only cross-connect linking OVH to our provider (in Montreal) is currently missing.

Wrong link, but everything is back to full speed no matter what :)

It's back up as of 16:50 EST (UTC-5)!

Not according to http://status.ovh.co.uk/?do=details&id=11304

Actually, as you can see at their network map[1], the cables are still out. Only the three new ones, upgraded today, are working.

[1] http://weathermap.ovh.net/#usa

I wonder, is this (still) common these days? Having a datacenter on a single physical cable?

To be fair, they're setting up the backup link. This DC, like many others, is well out of the way so running distinct fiber is lengthy process and even localizing fault and getting crews on site can be a drag.

The location of Beauharnois probably being chosen because there's a Hydro plant there with 1903 MW of installed capacity.

At Google -- no. At OVH -- yes, apparently. Google have it's own outages, caused mostly by a network config change or software update, but not from fiber cuts.

Plus Google has a shit ton of redundancy. I remember watching a video about their site reliability team. Their job is to test worst case scenarios and they can/will cut an entire datacenter out of a service's (gmail, drive, etc) pool to see what happens.

Their exercises are also excellent at including the full organization in DiRT, not just engineering.

eg, Just because someone is "authorized" to spend six-figures for scheduling a diesel delivery doesn't mean they will.

_Weathering the Unexpected_ https://queue.acm.org/detail.cfm?id=2371516

Reply on news.ycombinator.com