Hacker News Reader

Comment by nkcmr | Hacker News Reader

In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg

View on HN

I am in this DC, and we lost power to all of our racks but one. Power was restored about 2 hours later. I would assume Cloudflare had some significant failures in equipment due to the power drop. We lost a couple of servers that didn't come back up, which is not uncommon problem with hardware that has been running without a power-off for 4-5 years.

What I'm confused by is we had "power partially restored" 22 hours ago, and no news from PDX02 since.

I assume both Clouflare and Flexential are on DEFCON 1 right now, but I'm wondering if it might be more than just the building going dark.

There's something about a failover than was attempted and crashed halfway through, but unclear if that's what's causing the 24h+ situation.

if it was a state sponsored attack, various 3 letter agencies may want to investigate which could slow down restoration of service

If you can't cope with the loss of a data centre you're not really running a resilient system.

On yesterday's post someone that used to work at CF mentioned that PDX is "the brain" and if it goes down data stops propagating and starts getting stale. It's crazy to me that a company that is so critical to SO MUCH of the traffic on the internet doesn't even have a failover strategy for "the brain" of their operation.

Not great to know an aspiring hyperscaler is one tornado away from insolvency.

I dont think PDX would get many tornados BUT the PNW has the joys of earthquakes that are overdue.

It's amusing, I have plans on how much service degradation I can accept and what I need to keep working in the UK if the Thames Barrier is breached (which would flood most of the UK's internet connectivity in places like Telehouse, Sovereign House, etc)

To have 30% of the internet relying on a single building in a single city is hilarious.

Especially 15 hours after the DC comes back up and you're still offline.

Reply on news.ycombinator.com