BGP super-blunder: How Verizon sparked a ‘cascading catastrophic failure’

I was also interested to find out which one, given that I'm from Pittsburgh. Turns out that that this is what Allegheny Ludlum is called these days; they pay my grandpa's pension, and several other relatives worked for them for a very long time as well. And today, I work for Cloudflare, though not on any of the stuff that was involved here. It's a small world.

> I was interested to find out which steel mill, since they are almost non-existent in Pittsburgh these days

As evidenced by the smell of sulphur yesterday morning the coke plants down in Clairton are still alive and well!

Yeah, redundant lines are expensive.

Even multiple lines to the same DSLAM were considered expensive back in the day... so if you sliced a single line, you're SOL.

You'd essentially be DDoSed. The internet wouldn't work, and your firewall logs would show that you're being flooded with inbound traffic. The firewall would probably be dropping the unsolicited inbound traffic, so internally you shouldn't see any impact.

Unless as part of the misconfiguration that caused this, the firewall also thinks it knows how to route traffic to the affected prefixes, in which case it would be accepting the traffic and routing it, in which case segments of the internal LANs could be flooded too.

Part of this is Verizon's poor practices, and part of it is Verizon is a major transit provider; if a routing leak affects a large amount of the internet, it's almost certainly because a major transit provider accepted the announcement and propagated it.

Since they fail to filter these out, perhaps some enterprising person could route the entire internet through someone's closet ISP. Maybe, just maybe, they will then care.

Among the many ASes of things they've acquired, Verizon is AS701 / UUNet, which is historically one of the largest and most widespread ASes on the planet. I still think of it as uunet.

I remember setting up a T1 in the mid 90's. Our upstream was InternetMCI. There was no Web UI. There was no documentation of any sort. The guy (MCI tech) just had me read him routes over the phone and he announced them. Fun times!

easier explanation is that they were doing troubleshooting and had removed the filter to do that.

Ah, yeah the good old Internet factory : https://twitter.com/cryptonomatron/status/100839718221335347...

At least they tried to prepare for this situation. The software they were using was very close to the metal.

Forging the internet?

Oh the irony.