Docker Systems Status: Full Service Disruption

> as soon as people start using any vendor-specific functionality

It's also true in circumstances where things have the same name but act differently.

You'd be forgiven for believing that AWS IAM and GCP IAM are the same thing for example, but in GCP an IAM Role is simply a list of permissions that you can attach to an identity. In AWS an IAM Role is the identity itself.

Other examples; if you're coming from GCP, you'd be forgiven for thinking that Networks are regional in AWS, which will be annoying to fix later when you realise you need to create peering connections.

Oh and while default firewall rules are stateful on both, if you dive into more advanced network security, the way rules are applied and processed can have subtle differences. The inherent global nature of the GCP VPC means firewall rules, by default, apply across all regions within that VPC, which requires a different mindset than AWS where rules are scoped more tightly to the region/subnet.

There's like, hundreds of these little details.

I think there's some irony in Docker being impacted specifically, as they're one of the main tools to help achieve true multi-tenancy.

Multi cloud is just a way to have the outages of both.

And even if you think it’s important enough to justify the expense and complexity, it’s times like this when you discover some minor utility service 1) is a critical dependency, and 2) is not multi-cloud.

Complex systems are hard.

A bit like the ambiguity of search facets: if I select one facet, I get results that match, but if I add a second facet, should the results expand (OR'ing my selections) or contract (AND'ing my selections)? Presumably they should be OR'd if they belong to the same category (like selecting multiple colors, if any given result has only one color) but AND'd otherwise (like selecting a color and a size). But then a category could consist of miscellaneous features, and I want results that have every feature I've selected, which goes against the general case.

Almost all cloud providers help here by having inter-region failures as well.

There are multiple AWS services which are "global" in the sense that they are entirely hosted out of AWS East 1

During the last Docker Hub outage we found Google mirrors lost all image tags after a while. Image digest references would probably work

From what I've seen: I highly doubt it.

Edit to add: This might spur on a few more to start doing that, but people are quick to forget/prioritise other areas. If this keeps happening then it will change.

Yeah, perhaps. I don't know how many folks host mirrors. Most places I've worked for didn't, though this is anecdotal.

“Their own” can and often does mean something hosted on a major cloud provider (whether they manage it in-house or pay a vendor for their system)

Yea we have thought about that - I also want to remove most dependencies on externally imported actions on GitHub CI and probably just go back to simple bash scripts. Our actions are not that complicated and there is little benefit in using some external action to run ESLint than just run the command inside the action directly. Saves time and reduces dependencies - just need to get time to do that…

Hmm yea with a self hosted runner this could work. Gotta need to set the dockerd config into the VM before the runner starts I assume - unfortunately GitHub itself does not allow to change anything for the prepare stage - and it's a known issue for 2 years at least...

https://github.com/actions/runner-images/issues/1445#issueco... https://github.com/orgs/community/discussions/76636

I'll be honest and say I hadn't heard of Spegel before, and just read the landing page which says "Speed up container pulls and minimize downtime with a stateless peer-to-peer OCI registry mirror for efficient image distribution", so it isn't exactly clear you can use it for more things than container images.

What exactly does "stateless" mean in this context?

It's not entirely uncommon to only have parts of the solution open. So a license on one repo might not be the whole story and looking further would take more time than giving a good suggestion to the author.

Also it's good feedback for the developer of this solution

If I have to dig through your website/documentation to find basic information we’re not getting off to a great start. It’s pretty common for open source projects to proudly proclaim they are open source from the get-go. “____ is an open source tool for ______.” Simple as that

Thanks. I will keep an eye on your project as it looks great and something we would definitely benefit from.

P.S. Your blog could do with an rss feed ;). I will track https://github.com/spegel-org/spegel/releases.atom for now

Our images are in a private docker registry on quay.io

According to https://news.ycombinator.com/lists it's "Most active current discussions"

I find that it better surfaces the best discussion when there are multiple threads (like in this example), and it keeps showing slightly older threads for longer when there's still discussion happening.

More likely the outage reports for google and microsoft are based around systems which also include aws

In terms of user reports: Some users don't know what the hell is going on. This is a constant.

For instance: When there's a widespread Verizon cellular outage, sites like downdetector will show a spike in Verizon reports.

But such sites will also show a spike in AT&T and T-Mobile reports. Even though those latter networks are completely unaffected by Verizon's back-end issues, the graphs of user reports are consistently shaped the same for all 3 carriers.

This is just because some of the users doing the reporting have no clue.

So when the observation is "AWS is in outage and people are reporting issues at Google, and Microsoft," then the last two are often just factors of people being people and reporting the wrong thing.

(You're hanging out on HN, so there's very good certainty that you know what precisely what cell carrier you're using and also can discern the difference betwixt an Amazon, a Google, and a Microsoft. But lots of other people are not particularly adept at making these distinctions. It's normal and expected for some of them to be this way at all times.)

They might be using third party services that rely on AWS.

It’s very likely they’ve bought companies that were built on AWS and haven’t migrated to use their homegrown cloud platforms.