DE-Cix – Power failure leads to Downtime in germany | Hacker News Reader

DE-Cix – Power failure leads to Downtime in germany

heimatzeitung.de

18 pointsusernam3311 comments

Kinda fun to see the increase in AMS-IX traffic: https://ams-ix.net/technical/statistics

Keep in mind this is only one of the many DE-CIX datacenters, they exist in at least 8 datacenters in Frankfurt: https://twitter.com/goebelmeier/status/983453047543214081

Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage.

In fairness, we're in FRA8 but DE-CIX is not directly, so we're connected to their switches in FRA5.

Imagine it being 8pm, you sit in a operating room of germanys largest internet knot with up to 6,027 Tbit/s throughput and then the power goes down.

I am concerned that this even happened in a place important like this.

The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two hours (speculation).

Shouldn't we decentralize more?

Only those peers who are single-homed to Interxion FRA5 / DE-CIX6 will have experienced an outage. The whole of DE-CIX isn't down.

> Shouldn't we decentralize more?

> The whole of DE-CIX isn't down.

Largely agreed with you, though as general advice I would say that redundancy by adding more exchanges to your network serves you better than adding diverse connections to a single exchange.

Yep, that makes a lot of sense, especially since 10G/100G waves and such on the FLAP route are so inexpensive. Getting a remote peering port on AMS-IX or something is probably a good move.

> The article states internet was down for users at around 10pm (I expirenced it live). So they might have been on generators for two hours (speculation).

More likely, the timestamp in the picture is UTC, which corresponds to 22:00 CEST, cf. [0]

[0] http://status.core-backbone.com/index.php?id=1972

Well, now we know that significant portion of that facility are 1+0 for power systems. In a true 1+1 setup with air gap between A and B side power systems this wouldn't have happened.

We were hit by this.

In fact, we're hit by this again, down again at 22:09:19 EST for me and still down. Their website is broken too (https://www.de-cix.net/) and their email servers are offline which makes contacting them amusing.

Not much to be done, really, fortunately the impact for us is very trivial.

General home connectivity also suffered, at least a few people (and myself) had connectivity issues, if you search for @vodafone_de on twitter you'll see a lot of people complaining (though I assume it was not just vodafone with issues)

Just received an email update from INTERXION acknowledging they did indeed lose power again at FRA5. This leaves us in a bit of a mess.

Reply on news.ycombinator.com