Hacker News Reader

Comment by dgl | Hacker News Reader

Oracle also had issues around the same time: https://ocistatus.oraclecloud.com/#/incidents/ocid1.oraclecl...

Obviously the weather is easy to blame, but I wonder if the underlying cause is the same datacenter. It's kind of annoying when clouds use availability zones as such an opaque thing, it's not possible to map a zone from one cloud to another and potentially your failure domains overlap.

Shameless self promotion: I have a map of all the cloud regions (but not going into as much detail as availability zones): https://cloud-regions.bodge.cloud -- the clouds just don't publish the data.

View on HN

For the AZs that's slightly complicated by the fact that at least for AWS, they are randomised (your eu-west-1a isn't necessarily the same as mine), so it might be pretty much impossible to actually check for actual overlap even if you know one DC is used by both Google and Oracle. Spreading out regions seems more appropriate to increase redundancy.

Your own mappings are available in the console and on the API. So definitely still possible to tell

Amazon does provide a way to map to the real availability zone: https://docs.aws.amazon.com/ram/latest/userguide/working-wit...

It's also not appropriate in all cases to spread out, for example the nearest zones to London are around ~7ms away in mainland Europe.

The provided info is only relative to AWS. If you wanna map geographical destination across clouds you’re out of luck unless you have reports with precise location available. Generally, AWS already provides certain assurances that AZs are spread out.

I didn't know this, but it makes perfect sense and randomizing the a/b/c is a tried and true solution - e.g. in the electrical grid, L1/2/3 are rotated for each customer, because people tend to connect more stuff to L1 on average.

> people tend to connect more stuff to L1 on average.

I just plug into the wall socket and it works.

How could I choose to connect things to L2?

If your property has 3 phase power, chances are your single phase breakers are just on some random phase (picked by your electrician) ie phase 1.

Because phase 1 is picked so often, they actually rotate the phases at the street ie switch 1 2 3 with 3 1 2 to avoid having ie 300% more load on phase 1 and having the grid just die.

Hopefully not 123 -> 312 but e.g 231.

AFAIK phase order matters for e.g. three-phase motors, they will spin in the wrong direction if wired with the phases in the wrong order.

I'm not following, aren't 123,231 and 312 all in the same order?

312 is also a clockwise direction; the anticlockwise ones are 132 213 321.

Ours are called red, green and blue which helps avoid all on P1.

I wonder if there can still be a statistical discrepancy due to the electrician favorite color.

It seems blue is strongly favored by psychology students for example : https://www.livescience.com/34105-favorite-colors.html

UK Phases used to be coloured as well. The wiring regs changed to make it much less confusing (and cause fewer issues for colourblind people) so that they're now L1, L2, L3.

The GP is also a bit misleading. Most residential properties in the UK are only fed one phase - either the street only has one phase or the grid connects houses up L1, L2, L3 in sequence. Of course I'm speak about detached and semi-detached houses, in larger groups of houses (e.g. blocks of flats) you usually get 3 phases, and one of the jobs of the electrician is phase balancing.

I'm from Yeovil, Somerset! I have a fair idea about 'leccy here.

Then you'd know they're Red, Yellow and Blue, and that colouring went out in 2006:

https://www.newfound-energy.co.uk/electrical-three-phase-wir...

> If your property has 3 phase power, chances are your single phase breakers are just on some random phase (picked by your electrician) ie phase 1.

That's bollocks or speaks for a completely incompetent electrician. Usually, you have three phase rails coming out of your GFCI that distribute the phases sequentially to each breaker [1] to avoid that scenario, as well as ending up with a tripped main breaker because the customer overloads the phase.

The only case where great care is being taken to avoid random assignment of phases is in event stage technology - you do not want your lighting dimmer packs on the same phase as your amplifiers because dimmer packs inject extreme amounts of EM noise, and you also have to take care to not put overloads on any single phase anywhere. However, it's questionable on how long this will stay relevant, given that most lighting load is moving off to LEDs.

[1] https://www.amazon.de/-/en/Busch-Jaeger-Hage-Phase-KDN380A-3...

Guessing you’re from the UK where you seem to have fancy pre configured sub boards. In Australia the standard and 99% of house installs are just hand wired with single wires from breaker to breaker and they’re twisted together and rammed into the bottom of the breakers :)

Germany, and I've worked on construction sites and in stage tech for a time in my career.

What you describe is... beyond awful and an absurd fire risk.

Lmao I moved from NZ to UK and the house wiring here has very often not been touched anywhere from the 60s-80s or so.

They are very slowly modernising here, but they seem to have a cultural resistance to change/modernisation; they're in love with the old days.

Very different to Aus/NZ where we're eager to get the latest stuff/NZ is often used as a test market.

Most people only have single phase power here in the US with +120 and -120 legs separated by 180 degrees angle. The power company steps the voltage down from a single phase on the distribution wires. Circuit breaker panels are set up with alternating legs so that the neutral current is approximately balanced whether you fill up the panel on one side first or from top down.

You have some of the empty lat, lon data (e.g. AWS GovCloud (US)) interpreted as at (lat,lon) = (0,0), putting them incorrectly on Null Island.

Thanks, hidden.

UK infra - physical and electronic - is typically rated to around 30C. So higher temps are going to cause failures in multiple locations.

If yesterday's temps lasted for a week instead of a day or two a lot of essential stuff would stop working.

Carrier-grade (i.e. critical) network equipment should be rated for 40C ambient air and then up to 55C for some short length of time (3 days??) to allow for air conditioning failure. This is what we design and test for. Cheap stuff, including Google's software solutions running on 'commodity hardware', won't handle that. You get what you pay for.

Does the UK even design with the assumption of air conditioning being necessary and present, let alone designing for it failing?

Datacenters always have A/C I thought?

Thanks :)

Google and other companies do make risk assessments including temperature scenarios.

Yes. You must be able to control the temperature of a building densely packed full of hot radiators. You might be able to avoid active AC within the Arctic circle but you'd m still want to filter that incoming (very) cold air, at which point maybe you might as well basically have an AC system. And yes, you must ensure you can survive a failure because these systems do fail (normally when you need them most) and then otherwise everything inside cooks.

The arctic is warming much faster than other places. See eg this recent article about record 38C temps in siberia: https://www.sciencenews.org/article/siberia-verkhoyansk-reco...

Google doesn't run its networking equipment on commodity hardware. That's been the case for over a decade source from Open Network Summit talks. The issue here isn't about network equipment temperature rating, rather whole datacenter losing cooling.

I was under the impression that they still did some of their own software defined routing, maybe at lower bandwidths? That could be an out-dated view. I do know they also buy traditional equipment from the big vendors for high-bandwidth stuff. P4 is very cool.

I was trying to point out that high-end hardware should survive a couple of days despite a data centre losing its cooling. it is designed for exactly that situation.

a typical google data center has a networking room which often will have large numbers of standard commercial networking devices. That room has extra redundancy, locked off from the cattle, etc.

I have not seen a lot of carrier equipment in the DC/CO rated to 55C. Optical transport nodes maybe but not the Switches and Routers and servers etc. Temp ratings like that are normal for field systems in passively cooled cabinet shelters or tower sites and Industrial Ethernet and CCTV camera housings etc are even built to handle 65C. The Uptime Tier-1/2/3/4 builds overcapacity redundancy into the cooling facility itself to avoid failures like this and I don't recall seeing anything NEBS related to go beyond that either. This looks like it was a matter of intake air/water to the cooling system going outside of design parameters so even with extra capacity they couldn't meet the t/h set points.

I think it very much depends on the equipment. I think we have multiple tiers of what we support at each type of facility, with traditional data centres having less stringent requirements (maybe w.r.t noise/filters rather than temp) compared to the facilities housing big routers where -5/55C is the spec to meet.

You are right, and it's arguably harder for those smaller boxes sitting in cabinets / on telegraph poles, despite being much lower power systems to start with. It might be 50C+ in there before they even turn on! Those things might have a two-stage boot process where they just run their fans for a bit to cool things down before actually booting the main system. It must be a real nightmare for entirely passive stuff, I have no experience with that.

Cheap stuff, including Google's software solutions running on 'commodity hardware', won't handle that. You get what you pay for.

Sounds like the downside of the "pets versus cattle" methodology is that natural phenomenon will wipe out your entire herd, while the pets survive.

Why does your map have AWS's eu-north-1 in Estonia? Officially it's said to be in Stockholm and I can find no references nor know of it existing there.

Strange, I'm not sure where I got that from, will fix.

You have some locations simply pinned to the central squares of cities when that is definitely not where the facilities are. Maybe include a +/- error range on the provided data?

The eu-north-1 AZ’s are in Västerås, Eskilstuna and Katrineholm, all ~100 km from Stockholm and ~40-80 km from each other.

Wow, this map is really cool. I'm (very) idly curious how accurate the reporting for the Sydney region is, because I just looked up the lat/lon, and found myself in the middle of an urban mall environment that I've walked past many times when I've been in the city.

Being able to look up at the buildings there and know there are indeed 5 different clouds somewhere up above my head in that specific location would be really cool. Being able to point at a specific building would be even cooler.

I do of course (sadly) appreciate the flip side of this coin which is one of (many of) the reasons precise data is not published. So I guess I'm just wondering out loud, probably rhetorically :), how I might find out one day. "<-- That building" is enough resolution for me :)

EDIT: A quick Google found baxtel.com (among other websites) has address-level locations for most providers. The buildings are all so boring, haha! (Understandably so though.)

It’s pretty far off, at least for azure. It shows a number of azure regions as being located in downtown urban areas, which is typically not the case

It is possible to infer the locations if you had a lot of time.

AWS at least in the past use to co-locate with other third party servers in the data centre. And so you if you had such a server you could ping AWS endpoints to triangulate where physically those servers might be.

I'm pretty sure a majority of AWS AZs globally are still not in AWS-owned-and-operated facilities.

At this point I would be quite surprised if they were not majority AWS owned and operated.

Of the ones that I know details about in Europe and Asia, not a single one of them are AWS owned and operated. I'm sure the situation is different in the US, and wouldn't be surprised if they own and operate all of their US DCs.

I also think it would be nice if Google prominently stated which cloud regions are hosted in their real datacenters and which are in third party facilities. It's not just that you run the risk of some off-brand piece of junk overheating, but also because they so loudly trumpet their PUE and carbon neutrality, but then resell cloud capacity in other people's facilities where you have no idea what the carbon impact is.

That's not how carbon impact works. You don't buy "clean" products. You buy products from vendors who have an overall average mix of production that meats a "clean" threshold, and/or buys credits from someone else who exceeds threshold.

In Google's (and for instance Scaleway too), the point is that some of their DCs use renewable energy only, thus are "clean". If however you use the "wrong" Google region which is hosted in Equinix's DCs, which happen to be powered by coal, it's not even remotely close. However Google make the distinction and you can easily check which region is "clean" and which isn't.

They do tell you the carbon impact right on the region selector: https://cloud.google.com/blog/topics/sustainability/pick-the... and also https://cloud.withgoogle.com/region-picker/ is quite nice.

Google doesn't really have a concept of datacenters internally. In simplified terms everything runs on a borg cell, which usually but not always exists in a single building. Within a region one of your GCP services could be running on one cell, with another service running on another in a different building. If a failure happens or maintenance needs to be done, a cell could be drained to another one in the same region.

Google has "data centers" internally. People argue about the various definitions but a DC is usually one or a few buildings containing several clusters, each of which is fairly "close" to all its components.

Really, Google doesn't have an entire org called "dcops"?

Not in a way that is relevant to a cloud customer asking what building their jobs are running in.

Reply on news.ycombinator.com