Hacker News Reader

Hacker News

Comment by lukax | Hacker News Reader

Full thread

lukax

Google now has a "data lake" in Paris.

View on HN

This is not what I meant by digital ocean

This is what happens when your cloud condenses in the water/vapor cycle and returns to liquid form temporarily.

I don’t see the problem. Clouds are just water droplets anyway.

Joking asside I hope we will get a nice postmortem with juicy civil engineering details.

Google's DC is underwater

OVH's caught fire

What's next, us-east-1 gets hit by Godzilla?

Lol us-east-1 already went down for a day back in 2017 when an intern accidentally took down the whole DC. We could call him “Godzilla”

Source: my startup (stupidly) hosted our entire infra in us-east-1 at the time. Was a …tough day

us-east-1 is a great place to test your application resiliency :) it's like they threw in chaosmonkey for free!

It's funny because AWS, at least for the services I knew of when I was there, did rolling deploys to each region over several days. us-east-1 was always the final day because it was the biggest region, so you'd think it'd the safest region since everything getting deployed was well-tested. But while I was there I remember at least 2 COEs where the root cause was basically, "us-east-1 had some hacky legacy configuration that no other region has and that wasn't known/accounted for."

sounds like an easy day to me, like when power goes out, nothing to do anyway, esp. when the entire infra is there.

Some of the regions are so critical for AWS that them going down will bring down most of the control plane :P

if us-east-1 is not in Tokyo its safe

Earthquakes and wind damage should be next.

Water and fire already had their way, I suspect tornado and a landslide are next.

I hope whoever is hosting data in that zone has thoroughly tested and verified backups offline or with another cloud provider. Of course, you can complete DC failover, depending upon service needs, but it costs more resources. Either way, timely tested backups are the only way to survive natural or manufactured disasters. Good luck to Google OPs team and everyone else involved with the GCP region in the EU.

I'm down for coding to go full circle.

We called them bugs because you literally had to go in and get the dead bugs out of your electrical system.

Now we can call it fishing because some pirate has sailed onto your datalake and is looking for sunken hashes.

What do you think the hourly for Cloud Architect/Data lake power boy level 1 should start at?

Funny enough, Lucas Film suffered a same outage because their data center was backed up to the lake, and was basically under water, and then the wall started to leak.....

Do you have a source for that?

Yeah, me. I was one of the lead designers on the Lucas Presidio campus, including the DC there...

When we were still designing the site/systems/infra we would go to BigRock Ranch a lot for meetings and vendor interviews.

Its the datacenter at bigrock which has one wall which backups to the lake on site, and the DC is in the sub-level parking garage, and its rear wall was leaking due to the pressure from the lake on the wall.

Was it really an 'outage' or did they just turn things off while they plugged the leak?

You know what would be an interesting article would be to have a ton of outage/post=mortems etc.

So some of mine woule be

Chinese air usb gaps and phishing

What stories can you expound upon?

Is your data center suffering an outage due to flooding from a faulty wall you built against the man-made lake you built next to your $ Billion data center??

Call California and Meyers to see if you qualify for PG&E benefits.

Reply on news.ycombinator.com