Rogers letter to CRTC on cause of Canada-wide service outage of July 2022

crtc.gc.ca

26 points·deeesstoronto··7 comments

A lot of gaps not described in the letter:

Lack of independent management plane for core business assets. Driving to the office is not a management plane.

No external visibility of Rogers functionalities. The Kentik analysis of BGP and reachability suggests their tech staff had a lot of analysis to do before understanding of the problem.

No fallback plan in case of a misstep. No immediate detection of a misstep.

Four hour intervals in notifications (which implies lack of visibility into issues and absence of plans).

Secrecy regarding social and first-responder impacts (censorship of 911 call volumes for instance) suggests Rogers views these facts as existential risk, or at least of very high risk to their management.

Seems like most of the technical details are redacted :( This reads more like an obligatory corporate/legal response to an angry bureaucrat than a technical post-mortem.

Very light on details. The cause is discussed for 1 paragraph and then it's "Rogers understands... will work on..."

Of course, this is what happens when we privatize services. They get shittier on purpose as the owners extract as much value as profit for themselves as they can at the cost of everyone else

This is the whole point, of course they'll give the same old tired lines of improving and doing better while never actually investing in making their service improve

Non-doc version: https://preview.ninja/04b08c61c2a02e8255bbe268fe0e4049f7230b...

Related filings on outage with CRTC: https://crtc.gc.ca/otf/eng/2022/8000/c12-202203868.htm

That seems like A LOT of redaction.

Reply on news.ycombinator.com