“An ISP in Asia is leaking routes to a Tier 1 transit provider”

Really? That's not been my experience with L3 and a few others. They seem to prefer customer routes always, even when prepended. But I don't really understand this too in-depth.

The people managing peering for AS 3356 and AS 3549 should be the same group, no? ("That DB we don't share the URL of" seems to imply as much.)

But a route leak forces traffic to take a different path, and this potentially results in a pipe being inundated with a significant volume of traffic that cannot be handled by ISPs along that route (or simply a dead-end).

i.e. imagine an ISP in Rio suddenly declaring that they are the best route to reach the networks that contain facebook.com and google.com ... that ISP will DDoS itself or one of their downstream partners. An ISP may not even notice this, but their peers and partners probably will.

A couple of years ago a BGP route leak took out the entire internet for a few hours for people in Australia. They're fairly common unfortunately, but tend to have limited impact and are resolved quite quickly.

More info: https://labs.apnic.net/?p=139

https://blog.cloudflare.com/route-leak-incident-on-october-2... has more information from a previous time this happened. A key idea is that internet routing depends a lot on trust, and it's possible for a single misconfigured site to cause serious issues across the internet.

I got on with tech support this morning and they had no indication that anything was wrong regionally, had me do a factory reset on the device, and verified it was connecting to their network successfully.

So, maybe my issue is resolved, but Sprint is still having peering issues. That is the kind of behavior I would expect to see during an issue like this.

Of course I go into the store and they tell me there have been regional issues for the last two weeks and it's not my device (we can't help you), but everything works fine for me while I'm inside of the store and on the way back. Then 2 miles before back at the office, broken again, call again and phone support guy hasn't heard anything about this Malaysia Telekom debacle or any regional outages in my area. At this point I notice that I can still reach some sites, but most sites are down.

Not completely sure this is related, since I've been having issues since Wednesday afternoon (in fact that would seem to indicate it's unrelated). Also possible I have been having one issue that was fixed by the factory reset, so now I can freely experience the other issue that everyone else around the globe is apparently having today.

I am interested to know from The Expert as well, when Telekom Malaysia fixed their issue (it is actually fixed at the source now, right?) would there be some propagation delay or aftershocks, or should everything just return to normal relatively immediately.

In theory, a large portion of the Internet would have still seen the legitimate routes as better paths, as they would have a shorter AS path (fewer networks in between) than the leaked ones. However, many networks often implement other BGP metrics for traffic engineering, depending on whether they are seeing the routes from a transit/peer/downstream customer, which may override the shortest AS path.

Edit: It seems like that must have come through the Southern Cross Cable[0] which is nowhere near Malaysia.

0. https://en.wikipedia.org/wiki/Southern_Cross_Cable

Edit: found it. See Lecture 4: http://ocw.mit.edu/courses/electrical-engineering-and-comput...

So Telecom Malaysia messed up a config, and Global Crossing accepted their updates automatically.

Global Crossing didn't have to accept the bad update. They generally trust updates from other organisations that are generally trustworthy. They apply checks and restrictions proportionate to the risks involved.

These mistakes happen rarely. If they were to happen more often, major operators would apply more checks and restrictions. If they were to stop happening, operators would apply less checks and restrictions, because they have a cost in manpower, complexity, and loss of flexibility.

That's how the internet works. You could almost say that's what the internet is--the idea of being actively managed by people who know what they're doing and are not bound by exhaustive predefined policies is defining of how the internet came about and how it came to be dominant.

If you want a network guaranteed to be resistant to this kind of f---up, build one. The internet is that network which does not work that way, which is flexible, expandable, mostly "good enough" but not ever designed for absolute reliability.

Unlike in engineering sector where a negligence could result in injury or death; those could be charged.