Now Open – AWS Africa (Cape Town) Region | Hacker News Reader

Now Open – AWS Africa (Cape Town) Region

aws.amazon.com

239 pointsEwanToo83 comments

You might be wondering "why Cape Town"? when it is right at the bottom of Africa, and the Southern African business hub is around Johannesburg, 1262 km / 784 miles away, and the next area that needs it is Lagos, Nigeria ( 4760 km / 2958 miles).

However I think there are some upsides to Cape Town, i.e.

1) AWS already has team and infrastructure there in Cape Town. It is a bit of a tech hub as well.

2) Submarine cables land in Cape Town - and they cannot in Johannesburg as it is far inland. By length of internet cable, Cape Town is actually closer than it seems to other parts of Africa. (a) a lot of the connectivity is around the coast, not so much over land.

There's probably a shorter ping time between Lagos and Cape Town than there is between Lagos and Johannesburg, despite the map distances being the other way around.

3) The further north you go from Cape Town, the closer you are to data centres in Europe. Cape Town is closest to exactly those parts that are worst served by Europe.

I have no inside knowledge as to AWS chose Cape Town, but those seem like likely factors.

a) https://www.networkplatforms.co.za/information/our-network

Since this got popular and I can't edit any more:

1) Cape Town is not the Southernmost point of Africa, but it is very far south, and does contain the Cape of Good Hope, the Southwestern-most point. There are no population centres that are further south (Port Elizabeth is about the same latitude)

2) "...the next area that needs it is Lagos" ... and other central African cities such as Kinshasa and Accra. It might make sense if the second African AWS region is located hereabouts.

3) "... to data centres in Europe" ... and Middle-East, of course.

> I have no inside knowledge as to AWS chose Cape Town.

Well its public knowledge that Amazon has had a dev team in Cape Town for many years and they were part of the initial ECC efforts.

I fell asleep reading this article and had a sort of half dream about a post-apocalyptic game in which you the amnesiac protagonist (whose initials are “AZ”) gain consciousness aboard the MV Dirona, and to discover yourself must awake an ancient demon, a task that actually involves finding and activating long-lost AWS data centers, solving progressively harder puzzles using only in-game clues, a voice conferencing service over which a mysterious and cantankerous but also very tired-sounding voice feeds you tidbits of cryptic information in a mocking and self-righteous tone, and the gradually collected pieces of a 20,000 page wiki printout titled “Cold Start of AWS Region (confidential)” that is only about 80% accurate for each site.

Along the way you discover you are very good at playing yourself at board games, which is a key mechanic, and your rating improves with each region successfully started.

Final boss is us-east-1 of course, after which the identity of the “voice”, and the protagonist, are revealed.

Sort of like Myst, with an inverted Adam and Eve thing going on, and a lot more JSON.

This is a great step-up for companies using AWS here in SA.

On a personal note, my latency for my personal site will drop from ~170ms (eu-west-2) to ~3ms!

Edit: I misread, disregard this post

Is us-west really the closest alternative (by ping) to South Africa? Naively I would have expected one in Europe to be closer.

You may have misread, they used to use EU-west-2 (London).

Oops... you're right. Thanks.

They must have had to do extra work to cover for the failing South African electricity supply.

Awesome news as a South African. Guess Amazon's network blows away TENET? ;)

One thing to keep in mind is that each Availability Zone operates on independent power infrastructure, and each Availability Zone has independent power infrastructure!

https://aws.amazon.com/about-aws/global-infrastructure/regio...

James Hamilton [0][1] and others [2][3][4] have given pretty good overview over the years of the AWS infrastructure.

[0] AWS Innovation at Scale (2014), https://youtu.be/JIQETrFC_SQ

[1] Tuesday Night Live (2016), https://youtu.be/AyOAjFNPAbA

[2] Operation of the AWS Global Network (2019), https://youtu.be/UObQZ3R9_4c

[3] Foundations of AWS Infrastructure (2018), https://youtu.be/gH46jrFfiCc

[4] Exploring the AWS Global Network (2018), https://youtu.be/tPUl96EEFps

He's also written about AWS infrastructure (and electricity infrastructure in general) on his blog. Fascinating stuff.

https://perspectives.mvdirona.com/2017/04/how-many-data-cent...

https://perspectives.mvdirona.com/2017/12/when-you-cant-affo...

https://perspectives.mvdirona.com/2017/04/at-scale-rare-even...

https://perspectives.mvdirona.com/2018/11/1227/

https://perspectives.mvdirona.com/2019/02/aws-nitro-system/

I'd imagine they'd stick the datacenter somewhere that doesn't get regularly blackouts. I don't think you'd want to use generators that often

South African data center providers seem to have nailed reliable generator power - "...their systems are designed to operate continuously and effectively, regardless of whether they receive power from the national grid." [0] I don't want to imagine the cost.

On another note, load shedding (i.e. scheduled blackouts) have been implemented pretty uniformly fortunately, with areas following different published schedules - there's even an app for that! [1]

[0] https://mybroadband.co.za/news/cloud-hosting/336800-how-sout...

[1] https://sepush.co.za/

> Africa (Cape Town) is the 23rd AWS Region, and the first one in Africa. It is comprised of three Availability Zones

That means 3 data centres.

A common misconception, but no, AWS AZs are not DCs. A single AZ may be composed of multiple data centers[1], and a region may incorporate facilities that do not serve a public AZ[2], or that supply other capabilities[3].

[1] They'd be necessarily close together due to speed-of-light constraints.

[2] You may infer this from S3's triple-zone replication, which is still somehow magically fulfilled in regions that only have two public AZs.

[3] most obviously, Direct Connect.

To clarify, it means at least 3 DCs per AWS documentation.

https://aws.amazon.com/about-aws/global-infrastructure/regio...

Oh, very nice link, thanks!

> AZ’s are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.

This answers my previous question I guess

EDIT: well, actually, no. I wanted to know distance between DC in the same AZ :)

If you really want to know, you can look it up on wikileaks

https://wikileaks.org/amazon-atlas/map/

You're right, I should have said "at least three data centers"

It was in response to parent comment saying "the datacenter", which seems to be assuming that there was only one. But there will be multiple.

Do you happen to know how physically far can be AZs in the same region? The EBS sticking to the same AZ hints that they may be not that close, but I'm also very surprised to discover that 1 AZ != 1 DC

Most AZs are 1 DC, but some are too large to fit in a single building.

I don't think there is a distance limit between AZs but instead a latency limit. James Hamilton covered it in one of his keynotes a few years ago.

EBS sticking in the same AZ has nothing to do with distance and everything to do with keeping failures within an AZ

> I don't think there is a distance limit between AZs but instead a latency limit

Ok, fleshing that out via google:

"The connection between AZs is expected to typically achieve low single-digit millisecond latency between them on average."

https://knowledgebase.progress.com/articles/Article/expected...

and

1 light-millisecond = 299792.458 meters, so nearly 300 km / 186 miles.

https://www.translatorscafe.com/unit-converter/en-US/length/...

So in 2 milliseconds, light travels 599km or 372 miles.

That's a _hard_ limit. IIRC, modern communications is at a fraction of lightspeed, which would lower it.

There's no reason _not_ to have the AZs for a region in the same metro area, so separated by 10s of km, tops.

10km = 0.0333 light-milliseconds.

You need to take into account the speed of light in fiber optics and switching latency. For fiber you're looking at about 5 µs/km

> about 5 µs/km

Which puts it at 200km in a millisecond (5 microseconds is 1 / 200 of a millisecond.) Like I said "modern communications is at a fraction of lightspeed", although I didn't want to guess what fraction. But 200 / 600 = 1/3 is a reasonable fraction, seems legit.

> EBS sticking in the same AZ has nothing to do with distance and everything to do with keeping failures within an AZ

But they could be synced and if one AZ goes down another is elected as the primary copy. Most storage solutions handle that.

Quite. South Africa has a string of rolling black-outs though...some lasting as long as 8 hours at a time during bad spells. Some you'll be running off generators as primary source (with no backup) far more than you'd normally expect even if you have multiple data centres.

Easiest way around that is to locate it in an area exempt from the rolling blackouts as I said.

There hasn't been an 8 hour blackout in Cape Town due to loadshedding. The longest was 4 hrs I believe. They have a plan to get this sorted out in a year and a half, in the meantime DC's will just have to rely on their power redundancy (which has to be in place anyway).

Every time a new AWS region is made public, it continues to highlight the disparity in services availability across the regions, as well as making information about services available. Many regions are "discovered" because the ip-regions.json file is updated long before the press release, but it will be some weeks to months before key information needed to spin up infrastructure appears in documentation, for example things like the ELB hosted zone identifier, which at time of writing is not documented.

> things like the ELB hosted zone identifier

If you file a ticket on the proper page, I am sure that my colleagues will fix it ASAP. Go to the page and click "Provide feedback."

To extend what I'm trying to say is that customers shouldn't have to do this and that updating of this data (which ideally should be in a machine readable format, much like ip-ranges.json) is just another step. I would like to hope that AWS already has playbooks for taking a region out of closed-beta and making it GA. If the listing of af-south-1 is already present on other sections of documentation this may already be the case.

I'll send the feedback through anyhow.

Show me that magic unicorn company that spins data centres like that with 0 documentation bugs and I’ll join you criticising them. As it stands with limited resources I am pretty impressed with the speed AWS keeps adding capacity around the world.

Provide a service, or don't. It's sad that we've been trained to think that screw-ups are normal and OK.

Screw-ups will happen no matter how much wishful thinking and effort is spent preventing them. If you assume perfection in things then you will be sorely disappointed by most everything.

limited?

The AWS SDKs I've dissected contain the information needed to talk to each (service, region) tuple as machine-readable data. They seem to converge at boto:

https://github.com/boto/botocore/blob/master/botocore/data/e...

Updates somehow land automatically, e.g. this commit adding the new region a few hours ago:

https://github.com/boto/botocore/commit/f7dc4730ad34c6c3322d...

This is incredibly useful, thanks! Whist it has some region information and endpoints, boto's lacking other useful information - availability zone count, hosted zones IDs for services like S3, etc. This data publicly lives in a variety of tables across their documentation, and is painful to scrape.

Absolutely a fair criticism.

As a followup, I've found that the Terraform AWS provider needs information that boto doesn't – specifically, it needs region-specific details which can only be found in the documentation. They have a checklist for what to do when a new region gets announced:

https://github.com/terraform-providers/terraform-provider-aw...

Volunteers gather this information and push commits like:

https://github.com/terraform-providers/terraform-provider-aw...

I too am disappointed that AWS doesn't publish this information in a convenient way. On the other hand, well… source code _is_ a machine readable data format, and Go ASTs aren't that scary:

https://zupzup.org/go-ast-traversal/

Terraform follows AWS changes better than CloudFormation, so tracking Terraform might be a reasonable solution. One could even build a process to automatically retrieve the Terraform AWS provider source code, extract the necessary identifiers, and update the relevant data file living in an internal repository. Don't ask me how I know :-/

Terraform now has these identifiers:

https://github.com/terraform-providers/terraform-provider-aw...

Thanks to PatMyron at AWS, who apparently added this information to Terraform after seeing my mention above :-)

I know Epic Games uses AWS for their Fortnite matchmaking servers. And an African region has been a long-time player request. Maybe those wishes will be fulfilled soon.

Yeah lots of games leverage AWS. Solid win for gaming in Africa

It's a new data-centre. How does that have a bearing on the way Epic manage their matchmaking? Matchmaking isn't latency-sensitive, is it?

>Fortnite, one of the world’s most popular video games, runs nearly entirely on AWS, including its worldwide game-server fleet, backend services, databases, websites, and analytics pipeline and processing systems.

Granted, this is from 2018, but Chris is talking about how much they doing everything, and talking about how they like AWS' elasticity for their game servers because their load is pretty dynamic, and that they're running games in 26 availability zones.

https://aws.amazon.com/solutions/case-studies/EPICGames/

My South African cousins have desperately wanted an African Fortnite server. When playing on EU servers their latency was around 200ms on a good day iirc.

Oh, right. That's not a matchmaking server then, that's a dedicated multiplayer game server. Matchmaking is, to use Wikipedia's definition, the process of connecting players together for online play sessions.

in modern games matchmaking = game servers

which can be confusing

in a lot of games you cannot see a server list anymore, you can only use matchmaking (usually casual / ranked) and the matchmaking handles spinning up servers depending on the load.

They're still not the same service. Matchmaking is about grouping people together for the game, presumably dependent on their geographical location (for latency) and their skill-level (to balance the game), etc. Dedicated game servers, on the other hand, host the actual multiplayer session.

It's similar to the distinction Netflix uses between their control plane (hosted on AWS) and their content-delivery (hosted on their 'Open Connect' CDN).

Oh I agree on the terminology, and I think it's a shame things are how they are (no more active community on games like we see with 2000s games). I was just pointing out that it was so common these days that people mean "game servers" when they say "matchmaking".

Well you could also have matchmaking servers and P2P gameplay without any dedicated servers, which is popular with console games.

You're right P2P is more common on game consoles (Almost if not all Nintendo multiplayer games use P2P, Mario Kart, Splatoon, Animal Crossing, etc...) which in my opinion was 'okay' for a free service but a joke now for a paid one.

I might have expressed myself badly in my previous comment, I simply wanted to say that I've seen a lot of people saying "matchmaking" for "game servers" these days because it is very common and dedicated servers are no longer the norm (sadly).

Awesome news, especially for gaming on the continent. I won't lie, AWS Cape Town is kind of a dream gig for African technologists.

No it's not, if I'm based in Tunisia for example, why would I pick that region over one in Europe?

I perhaps should have qualified this with Sub Saharan Africa. Tunisia and the rest of North Africa might as well culturally be another world.

Tunisia is practically Europe.

That's my point. It makes no sense to talk about "Africa".

Well then you would be based in Europe, thus ruling out the original qualifier.

Of the big public cloud providers, Microsoft came to Africa first.

As far as I know, GCP doesn’t have anything on the roadmap for Africa.

I don't think they are playing the game of "Who is first attacking a market". I bet some Azure clients are happy to move to AWS if they were forced to use Azure due to being the only one. I bet also the other way around.

South Africa has a huge .NET market.

My speculation is that Azure was in ZA first though because of Office 365 however!

Why would GCP expand when they are looking to kill the product offering?

There is a lot of snark surrounding GCP. I'll just leave this here:

https://cloud.google.com/blog/topics/infrastructure/new-goog...

This year alone GCP will launch new regions in Delhi (India), Doha (Qatar), Melbourne (Australia) and Toronto (Canada).

Can you elaborate on this?

I've never heard before that they are trying to kill GCP?

Perhaps they are basing it on this [1] ultimatum supposedly issued by GCP’s top brass to best the top 2 cloud providers by a certain date. It was widely discussed on HN some months ago.

1: https://news.ycombinator.com/item?id=21815260

Yeah that conversation was pretty silly and more HN echo chamber because folks are mad that Google Reader was shutdown. Below is what I commented a few months ago re: this topic.

> Google is currently building a massive $500 million datacenter outside of Reno as we speak, and has 10+ billion invested in their datacenter cloud offering buildout this year alone.

> https://9to5google.com/2019/07/01/google-data-center-nevada/

> https://www.rgj.com/story/money/business/2019/02/13/google-i....

Could you elaborate on this/provide a source?

The AWS EC2 service was invented in Cape Town, AWS already have a lab and infrastructure there so its easier for them to scale up infrastructure when the people, skills and local ecosystem is already present.

I hope they will consider creating a new region also in Lagos, Nigeria. It's the booming[1] megalopolis and biggest city in Africa.

[1] https://www.brinknews.com/this-country-recently-became-afric...

Azure bandwidth out of their Africa datacenters is $0.181 per GB, while AWS is $0.154 per GB, will be interesting to see if this forces Azure prices down.

From memory the initial development on AWS was actually done in South Africa. Yup, here we go... https://en.m.wikipedia.org/wiki/Timeline_of_Amazon_Web_Servi...

Also

https://news.ycombinator.com/item?id=22943445

and

https://news.ycombinator.com/item?id=22943674

Cape Town is becoming a serious tech hub

Fwiw, EC2 was founded in Cape Town and continues to have a big presence to date. Later, the founding team built Nimbula, kind of DC/OS of the time.

https://www.theregister.co.uk/2010/06/25/nimbula_cloud_os/

Also see: http://www.adccpt.com/#/challenges

This video was shot in 2016 (at EC2s 10 year anniversary): https://www.youtube.com/watch?v=9Gk_I_0eMDA ... almost all of the folks in this video are still there.

Cool! Had no clue that it was.

https://en.wikipedia.org/wiki/Amazon_Elastic_Compute_Cloud#H...

The source is dead. I thought wikipedia would be better about that

> The source is dead. I thought wikipedia would be better about that

You could have fixed it instead of writing such comment. Wikipedia is nothing without the thousands of volunteers writing (and fixing!) content on it.

I would have expected they open up one in Lagos.

I wonder if this will improve international connectivity times once Cape Town's Cloudfront is warmed up since the underseas cables wouldn't need to carry as many CDNed assets due to new region presence.

Does anyone know how this will affect the water issues that Cape Town have been having? As far as I know, Cape Town's 'Zero Day' is still something people are cognizant of.

Not really actually. The drought ended. It does go in some regular cycles if you look at the historical dam water levels. Currently were out of the bad times, but efforts need to be scaled up to be ready for next time. Politically speaking this won't happen though.

In terms of climate, CPT is basically a more windy Los Angeles/SF.

Nice. Some competition for Azure.

Ooo I wonder if this will be the standard for African EHR and PHI

Reply on news.ycombinator.com