A coding error caused Rogers outage that left millions without service

In Canada 911 service is accessible without a SIM card; if your phone doesn't have a SIM, any cell tower should in theory still accept and route the call as normal. However in Roger's case, because the cell towers and their authentication mechanisms were still operational, any 911 call with a Roger's SIM card in it would route through Roger's network and only Roger's network; the one that couldn't service any calls. In essence, your 911 service was completely cut off.

The workaround is to pull the SIM card prior to making the 911 call, but it leaves an interesting question about what you're supposed to do in an eSIM world where pulling the SIM is not possible but you're again in this kind of situation.

To be clear I don't know if this is a real problem or not, but it is an interesting thought either way.

This is an interbank network which is essentially a cooperative operated by the country's major financial institutions.

Who would have thought Interac would've only had one ISP? Thankfully, they've stated they're adding at least one additional service provider.

I drove from Winnipeg to Calgary that day (a 12 hour drive) and had to stop in several small towns with TDs to pull cash to fill up.

It was quite bizarre checking Twitter all day to see if it was back yet and seeing it was still down.

It sounds like their control/management plane (with the user's database) was dependent on their data plane. So a data plane outage was more challenging to mitigate than it should have been in a decoupled architecture. Good lesson for any architecture.

Lots of things went wrong here, to name a few:

- lack of rigour in their process that allowed a critical modification without understanding its effects in context

- lack of monitoring and metrics to alert them immediately of the problem

- lack of emergency rollback capability to revert back to the known-good config

- lack of independent business-continuity comms channels. if you're a Rogers CTO or senior responsible adult and you don't have a backup Telus or Bell SIM card (and vice versa), you learned why you need one.

[source - 20 years of NetSec and NetEng experience]

And all through the day I kept thinking to myself "I bet someone pushed an update to prod, causing this. And I'm glad that this time it wasn't me."

> But, at 4:43 a.m. on July 8, a piece of code was introduced that deleted a routing filter. In telecom networks, packets of data are guided and directed by devices called routers, and filters prevent those routers from becoming overwhelmed, by limiting the number of possible routes that are presented to them.

> Deleting the filter caused all possible routes to the internet to pass through the routers, resulting in several of the devices exceeding their memory and processing capacities. This caused the core network to shut down.

Lesson no 1: Do not design your system to have a single point of failure.

Seems like an innocent and unavoidable mistake.

My opinion: You know how Juniper routers require you to commit a change? That should be enforced by a dedicated firmware and there should be multiple rollback confirmations (optional). So you commit a change and confirm, an hour later you have to confirm again else a hard reset and rollback takes place with a protocol in place where the rollback can be delayed when other devices in connected segments are also doing a rollback.

I don't get this. If a switch can handle only x number of routes, it won't get "overwhelmed" if you try to add more, it would just refuse to accept any more routes. Why would the network go down completely? These things are designed with extremely robust software, you simply start getting error logs saying that the device has run out of memory, but the operation continues. Most of the heavy lifting is done in hardware chips, so the software just can't program new routes, but the routing itself should continue unaffected.

https://www.theglobeandmail.com/business/article-how-a-codin...

The Outage The configuration change deleted a routing filter and allowed for all possible routes to the Internet to pass through the routers. As a result, the routers immediately began propagating abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their capacity levels and were then unable to route traffic, causing the common core network to stop processing traffic. As a result, the Rogers network lost connectivity to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers.

The Recovery To resolve the outage, the Rogers Network Team assembled in and around our Network Operations Centre (“NOC”) and re-established access to the IP network. They then started the detailed process of determining the source of the outage, leading to identifying the three Distribution Routers as the cause. Once determined, the team then began the process of restarting all the Internet Gateway, Core and Distribution Routers in a controlled manner to establish connectivity to our wireless (including 9-1-1), enterprise and cable networks which deliver voice, video and data connectivity to our customers. Service was slowly restored, starting in the afternoon and continuing over the evening. Although Rogers continued to experience some instability issues over the weekend that did impact some customers, the network had effectively recovered by Friday night.

what was the root cause of the outage (including what processes, procedures or safeguards failed to prevent the outage, such as planned redundancy or patch upgrade validation procedures);

Like many large Telecommunications Services Providers (“TSPs”), Rogers uses a common core network, essentially one IP network infrastructure, that supports all wireless, wireline and enterprise services. The common core is the brain of the network that receives, processes, transmits and connects all Internet, voice, data and TV traffic for our customers.

Again, similar to other TSPs around the world, Rogers uses a mixed vendor core network consisting of IP routing equipment from multiple tier one manufacturers. This is a common industry practice as different manufacturers have different strengths in routing equipment for Internet gateway, core and distribution routing. Specifically, the two IP routing vendors Rogers uses have their own design and approaches to managing routing traffic and to protect their equipment from being overwhelmed. In the Rogers network, one IP routing manufacturer uses a design that limits the number of routes that are presented by the Distribution Routers to the core routers. The other IP routing vendor relies on controls at its core routers. The impact of these differences in equipment design and protocols are at the heart of the outage that Rogers experienced.

The Rogers outage on July 8, 2022, was unprecedented. As discussed in the previous response, it resulted during a routing configuration change to three Distribution Routers in our common core network. Unfortunately, the configuration change deleted a routing filter and allowed for all possible routes to the Internet to be distributed; the routers then propagated abnormally high volumes of routes throughout the core network. Certain network routing equipment became flooded, exceeded their memory and processing capacity and were then unable to route and process traffic, causing the common core network to shut down. As a result, the Rogers network lost connectivity internally and to the Internet for all incoming and outgoing traffic for both the wireless and wireline networks for our consumer and business customers.

how did the outage impact Rogers’ own staff and their ability to determine the cause of the outage and restore services;

At the early stage of the outage, many Rogers’ network employees were impacted and could not connect to our IT and network systems. This impeded initial triage and restoration efforts as teams needed to travel to centralized locations where management network access was established. To complicate matters further, the loss of access to our VPN system to our core network nodes affected our timely ability to begin identifying the trouble and, hence, delayed the restoral efforts.

Despite these hurdles, our preestablished business continuity plans enabled staff to converge at specific rally points. Those equipped with emergency SIMs on alternate carriers that enabled our teams to switch carriers and assist in the initial coordination efforts. Further, we rapidly relocated our employees to two of our main offices in the GTA (# #). The critical network employees were able to gain physical access to our network equipment. Other essential employees were able to use alternate SIM cards, as per our “Alternate Carrier SIM Card Program” (described in Rogers(CRTC)11July2022-1.xiii below). Other employees were able to work from # #. Together, these groups were able to establish the necessary team to identify the cause of the outage and recover the network.

what contingencies, if any, did Rogers have in place to ensure that its staff could communicate with each other particularly in the early hours of the outage;

On July 17th, 2015, the Canadian Telecom Resiliency Working Group (“CTRWG”), formerly called Canadian Telecom Emergency Preparedness Association, established reciprocal agreements between Rogers and Bell, and between Rogers and TELUS, to exchange alternate carrier SIM cards in support of Business Continuity. This is to allow TSPs to communicate within their organizations in the event of loss of their respective networks. Bell, Rogers and TELUS took the lead to provide SIM cards to all CTRWG members.

# #.

When it was realized that Rogers entire core network was offline, employees started swapping out our Rogers SIM cards with our alternate carrier SIM Cards. This previously established contingency plan allowed us to begin communicating within our organization in the early hours of the outage and to start restoring services.

i feel like there's a lot of passing the buck in the video

Let's say this was a mistake. When they tested the config in a pre-prod environment, they would have noticed the redistribution of those routes. If they were pasting the config into a router instead of sending it via scp, maybe there's a mouse buffer/paste error, but you don't do that on a core device anymore. Judging by the response time, they weren't doing this onsite at the console either and their OOB access didn't work because it probably used the bunged cell network. Not negligent, but certainly risky.

I have personally made routing mistakes that caused peer resets, like mistyping a prefix on a static route that overrode the route to a peer interface over an exchange, and I have also overloaded circuits because of misguided path prepends. I haven't had enable privs on a core router in probably 20 years, but this wholesale redistribution is caused when you take one peers routes and announce them to another with you as the origin, known as "announcing the internet." As I remember, this isn't a single filter.

Not to muddy it, and it doesn't help my attack theory much, and please call this out as bullshit with a correction if you recognize it to be as such, but someone who used to work there told me something to the effect of their architecture had combined some of the older ASNs from previous Rogers acquisitions and that was part of the problem, whereby they essentially have the old AS's speaking iBGP to each other internally within Rogers, with only the single Rogers AS facing the internet as a peer, sort of the way the some of the old mega ASNs. It implies there were different policy domains with different levels of maturity inside the main AS, and those internal peers don't filter routes they recieved from the core. When you have a network that has grown organically over the last couple of decades, this happening at least once is inevitable. Legacy policies just redistributed what they were sent, and between flapping, dampening penalties, transiting and backhauling mobile voice traffic over IP networks, and backup and OOB reachability, this one of those perfect storms where for years the network both worked magically, and then one day failed just as magically.

Undoubtedly, there are probably some senior engineers sitting around saying, "I told you this would happen, and we've been trying to tell you for years," but when you know this stuff, it's your responsibility to also be persuasive.

I was very attached to the attack theory for a bunch of reasons, but the conceptual chain of events above makes more sense to me, and it is a lot more like a rubber gasket on a fuel tank expanding too rapidly an unexpectedly cool day. Obvious in hindsight and simple to explain, but invisible up until the moment of ignition.

When the human waste hits the circular cooler device, that's when we find out how well an organization is built and managed.

Who is surprised here it took them this long to recover? The necessary no-blame culture, temporary removal of decision barriers, all-hands-on-deck to get back to normal, all the things the lucky of us take for granted are guaranteed to be missing here.