Analysis of Today's CenturyLink/Level(3) Outage

If that is the case, there are a couple of reasonable ways to handle this.

One is to bring back something that used to be common in high level chess: the adjournment.

This used to be common when most championships used time controls that were slow enough that you often would not finish the game in one play session.

The way it worked is that at the end of the play session, the arbiter calls for an adjournment. When the player on the move decides their next move they write it down on a piece of paper rather than actually making it on the board, the move is sealed in an envelope which is kept by the arbiter, and play stops.

When it is time to resume, the arbiter opens the envelope and plays the sealed moved on the board and starts the clocks.

Both players are free to analyze as much as they want between play sessions, and can get outside help. For world championships, especially back when it was Fischer vs. Spassky or Korchnoi vs. Karpov and the match was serving as a proxy for the cold war, each player would have whole teams of top GMs to help analyze during an adjournment.

The player who sealed the move has the advantage of knowing for certain what position will be on the board when the game resumes. On the other hand, the other player is going to be the first one to have the move after several GMs have pulled an all-nighter analyzing it for them so any inaccuracy in the sealed move is much more likely to be punished than it would have been without adjournment.

Or get a freaking modem. Exchanging chess moves does not require high bandwidth, and most of these internet outages do not take out phone service.

Do a mini-adjournment (seal the move, but keep the players on site), establish a dial-up connection between the playing sites, and then unseal the move and resume.

While reading this I got impression they were genuinely trying to tone down the mud slinging they normally do while also trying to make it clear the outage wasn’t their fault. They just need more practice.

A provider as large as Cloudflare will always be impacted by other providers. Hopefully that point is clear to them now. The worst thing that can happen is they get a reputation they can’t play ball nicely and their peers and partners get tired of them. Service to their customers will erode over time because those peers and partners will screw with CF behind the scenes. It’s better to have friends than enemies with the type of business CF is in.

I would expect Google to have a similar explanation if a significant number of GCP customers faced an outage.

You should know, it wasn't just someone else's network that went down, that network brought down a big chunk of the internet with it. I think technical honesty comes before political appearances. The #hugops and mention of their past experience with a flowspec outage is clearly there to signal that the blogpost is not there for blaming or making L3 look bad.

On a similar scale I think a "scientific" observer of nature might say one shouldn't worry about local maximums because there is always some input stimuli to disrupt seemingly stable long term maxima, and that measurable dynamism of categorical evolution comes through dipping to local mins, and finding a new path to the new temp max because the topology of the surface changed.

Humanity challenge I think is in recognizing that almost all successful solutions to long term problems is built from small functional systems that scaled up rather than top down predictive and prescribed methodologies (mostly accidents until recent history). We can stand in nature and appreciate the diverse and varied approaches nature has taken to solving the particular localized problem (soil, land topology, etc...) but we don't evaluate the suffering and time frame it might of taken to find that particular solution. The plants that were either prematurely stunted or culled are just as much part of the ecosystem and fertilizer that holds the visible flora.

While appreciable suffering can and should be addressed and planned for, localized short term solutions are the things that take hold and spread until the collapse under their own lack of structural stability. Luckily nature is always at the ready to continue the experiment. Trail blazing is not about clearing the path, but finding it, the shortest path for nature is almost always the most preferred, you could say many of the foundations of physics[0] are rooted in it

[0]https://en.wikipedia.org/wiki/Fermat%27s_principle

It is not that we have built the Internet despite BGP. We have built it thanks to BGP. If we didn't have BGP we would have to invent it :)

There is this tendency in startup bros to assume BGP must be bad because it's old but in general BGP still holds up as a great protocol and is still something people regularly deploy into shiny new systems internally (e.g. bgp evpn vxlan) with no hesitation or regret, which is what L3 was doing here. The only real fundamental problem with BGP is that the assumption that "the internet" is such an internal system no longer holds. Solving that is primarily a social and economic problem in the same way other internet wide protocol upgrades are, just with the added difficulty of convincing a bean counter you need a few 100k for new routers.

It's tough because of path dependence. If you don't survive the short term problem, the long term doesn't matter.

your definition of civilizational collapse differs a bit from mine. i'm not altogether certain which one is more useful.

Of that, and a Debian mirror, and maybe the top ten-thousand github repos or something...

It was a head scratcher until this mornings news.

CenturyLink “business” is everywhere in CA and a large provider.