kemals

ThousandEyes observed the outage from multiple vantage points deployed behind Starlink: https://www.linkedin.com/posts/thousandeyes_thousandeyes-dat...

ThousandEyes analysis of the outage: https://www.thousandeyes.com/blog/zoom-outage-analysis-april...

This was a rather interesting event. In general, changing the IP address (even the loopback address) shouldn't have caused it from the BGP perspective. For example, if you were to change the IP address of BGP enabled router that has multiple BGP sessions, all other routers tore down the sessions to it, and withdrew the prefixes. BGP reconverge events take time. However, less than this took (90+ minutes and then a few more hours until __full__ recovery).

This seems like one of the events in which they changed IP on Route Reflector routers that were pretty busy, which would cause reconvergence and CPU spikes for all routers that it had sessions with. Also, there was a lot of volatility, as part of which re-advertisements were happening continuously. They also attempted rollback, which caused reverse operation, which triggered reconvergence. The other scenario is doing this change on the SDN controller, which affected all other routers.

More details: https://www.thousandeyes.com/blog/microsoft-outage-analysis-... https://www.thousandeyes.com/resources/na-microsoft-outage-a...

ThousandEyes public outage map shows the scale of the Office365 outage: https://www.thousandeyes.com/outages/

If there was a shorter route and you took longer one you are dealing with suboptimal routing :)

However, "is a bit sad in a way" part of sentence is interesting one. Edge services hosted within AWS/Cloudflare/Akamai improved customer experience significantly given that waiting for trans-Atlantic or trans-Pacific latencies is not thing any more.

This comment is spot on. This is asynchronous nature of the computer networks. While it is easy to control the path within smaller or enterprise networks, it is very likely that the reverse path on the internet is going to be different compared to forwarding path. Following that, the real challenge is, in case that you are dealing with some issue, to detect where the issue is, given that it might be happening on the reverse path that you don't have visibility into.

Thanks for providing quite good context in the post itself!

I was just about to ask about differences with Tailscale, which is solving a lot of the challenges outlined in the post, but you answered it:

"First off, Netmaker is super fast because it can use kernel WireGuard. There are some other WireGuard-based solutions like Tailscale, but they use userspace WireGuard, which is much, much slower."

Good luck!

Here is The Internet Report episode on the topic of recent AWS outages that covers outage and root causes: https://youtu.be/N68pQy8r1DI

The outage is clearly visible on the Internet Outages Map provided by ThousandEyes: https://www.thousandeyes.com/outages/

Not sure about #1 but regarding #2 it seems they never reached back to Cloudflare (based on the Tweet from Cloudflare’s CEO just yesterday).

Here is how ThousandEyes viewed the impact: https://twitter.com/thousandeyes/status/1146862826566250499

Companies and organizations are de-aggregating larger space into multiple /24 (most often) to extend the usage of their v4 IP address space, as nicely covered in this article: https://labs.ripe.net/Members/stephen_strowes/visibility-of-...