Rogers networks reliability and resiliency assessment after 2022-07-08 outage
crtc.gc.ca
crtc.gc.ca
It’s a shame on Canada and on Canadians that foreign competitors are still prohibited from coming into the market.
Allowing Rogers to purchase Shaw was absurd.
The Commissioner of Competition was very strongly opposed to the merger, the court ruled against him, and Canadians had to pay Rogers $13 million CAD because he opposed it.
I was a Shaw Mobile (not Freedom) customer, and Rogers wanted me to drive 600km (both ways) through the Kootenay mountains in winter to get my new SIM cards. I told them where to go.
Rogers is junk and Canada is an oligarchy.
We've been on a waiting list for a doctor ever since. My only option to see a doctor here is to go the emergency room at the closest hospital.
Last year I started feeling exhausted 24/7 and tried to make an appointment, but got turned away from my doctors office because my doctor retired and the new guy didn't want to let me make an appointment.
So here I am on a waiting list, in a city where 10% of the population don't have a doctor either. I've been waiting for over a year.
I wish I could change fields, "once you've seen how the sausage is made, you'll want to be a vegetarian"...
I wish our gov't served us better, and I try my best to enact change, but it's a fool who thinks any country is without its faults.
And I've lived in a few different countries in my life. No country is without faults, but, Canada right now is a mess.
Jokes on me. ;)
I love the land and the people, but, the country itself is a mess. We have already planned our exit.
Western Europe has all those things, is actually welcoming, has sane healthcare, and you don't even need to own a gun!
Are you in software development?
I was a Shaw Mobile (not Freedom) customer, and Rogers wanted me to drive 600km (both ways) through the Kootenay mountains in winter to get my new SIM cards.
Weird. They mailed my new SIM cards to me.
Rogers support is pretty notorious country wide, I'm sure it still varies at a localized level.
I had a Rogers internet install technician come to my house (for Teksavvy because the gov mandates they run the networks for small operators) and instead of drilling a hole through a wall he opened a window a crack ran the cable through and went home. The girl on the phone couldn't believe my story but I'm sure nothing happened in terms of punishment.
This is the outcome of artificial barriers to competition in the name of national benefit. It doesn't matter how absurd your customer story, neither customers nor the upstart businesses have a choice.
They could also have just eliminated our archaic foreign ownership laws. I am surprised this hasn’t come up in trade negotiations. If we want improved productivity then we desperately need competition. If we don’t get serious about this then we’ll continue to languish economically as a nation.
Their system didn't work when setting them up, and I had two different support operators tell me that my only option was to drive to Cranbrook. In the middle of winter. This is a drive that involves 3 mountain passes.
?
They had $10b+ in equity to work with and had ample opportunity to capture ZIRP if they wanted.
Anything inhibiting Shaw’s ability to raise capital was its dual class structure where the Shaw family retained control despite owning a minority of the shares.
They wanted me to drive to the Rogers store in Cranbrook or Kelowna.
There is some foreign competition via roaming, e.g. both T-Mobile and AT&T have US mobile postpaid plans with unlimited Canadian roaming, https://www.att.com/international/canada-roaming/ & https://www.t-mobile.com/cell-phone-plans/use-your-phone-in-...
For devices with low data (text/email/web) usage, there are vendors of global prepaid eSIMs that last 6-24 months, which roam on Bell and Rogers 5G networks, priced around $5/GB, e.g. Eskimo (Singapore) and AIS (Thailand) eSIMs.
These aren’t real options as anyone calling you back will be forced to pay long distance.
Eskimo (Singapore telco) has both North America and Global eSIMs.
Those don't seem that much cheaper compared to local competitors. For instance the cheapest plan at&t offers is $30 USD/month, but public mobile is offering a 40GB unlimited talk text plan for $29 CAD/month ($21 USD).
No, no it isn't a shame, it's great.
Preventing domestic mergers, and ensuring domestic competition exists are viable alternatives.
With viable alternatives available, when it comes to domestic telco and networking, you want domestic control. A nation is benefited by domestic ownership of its own infrastructure.
This is inline with domestic food production, manufacturing of defense products, transportation infrastructure, and on and on.
Domestic control is an aspect of sovereignty.
Now again, I want competition. The problem isn't foreign competition, it's a lack of domestic.
The Harper Conservatives worked on this, limiting spectrum auction to induce competition and so on.
I suspect someone had a pet peeve, re domestic cost of services. One thing about government, is that often politicians get into the politicking business because something pissed them off.
So I suspect someone was cheesed at costs, or customer service, and made it their thing. Sadly, it was dropped by the next government.
Another example would be the Trudeau Liberals overhauling the entire ombudsman process for banks, along with intense investigation into retail banking processes.
Prior, the ombudsman was an escalation process that ended up at a provincial ombudsman's office... jointly owned by the banks!
Again, I expect someone in the current government was cheesed at banking chicanery.
Regardless, we can get competition without foreign competitors. It just takes a government caring enough.
And it would take just as much push to drop foreign ownership requirements, as to limit spectrum, prevent mergers, etc.
— cross pollination of talent, knowledge, technology, etc;
— less susceptible to being eliminated when a future government caves on some of the rules you’ve mentioned
When the patriot act was passed in the US, suddenly some Canadian credit card holders, whos companies which had processing centers in the US, had all of their transaction info shared with US homeland.
If the corporate entity is domestic, foreign powers won't have control over their domestic corps, managing foreign infra.
Yeah during NAFTA 2, Canadians were all for protecting high prices and very little choice just to show Trump and the USA that they could take their cheap prices and value and shove it.
Rogers, Bell and Telus went on and made Trudeau classify them as Canadian media companies so they would never have any competition from the USA.
I guess Canadians like high prices and little choice..
ooops. that's actually quite funny
in case you didn't know, Rogers is one of canada's "big 3" telecom providers. The outage in 2022 basically crippled our economy for a couple days (most ATMs / interac didn't work)
This is networking 101. Heck, this is engineering 101. The real question is how a network provider as large as Rogers managed to be so poorly engineered in the first place.
Rogers isn't the massive network it is because Canadian consumers love them and want to use them - they just have no choice, and the government does all it can to keep it that way
Why spend money on reliability when there are no consequences of not doing so?
At this point, the government can fine them but it’s just passed on to the consumers. Maybe a random regulatory fee. Can’t close them down otherwise risk massive downtime to critical infra (emergency services)
And anyone who is good at technology generally moves across the border for the much better wages--and actually being able to work at a real tech company.
In short: brain drain is a factor too.
The ghosts of Northern Telecom, Blackberry and Instant Pot (ex-Nortel) would like a word.
Also Bombardier aircraft and "AI".
https://www.utoronto.ca/news/u-t-deep-learning-pioneer-geoff...
> When you translate a sentence using Google, or ask Siri to send a text, or play a song recommended by Spotify, you are using a technology that owes much to the innovative research of Geoffrey Hinton.. “deep learning” – a form of artificial intelligence (AI) based on neural networks.. Hinton’s revolutionary contributions to the field have earned him the nickname “the godfather of deep learning,” and have made Canada a hotbed for high tech.. for his excellence as a global pioneer in deep learning, Hinton received a Doctor of Science, honoris causa from the University of Toronto, where he is a University Professor Emeritus.
Also, bringing up Blackberry to make Canada seem relevant kind of has the opposite effect.
The topic of this subthread is not relevance, it's engineering culture.
Canada produced a device that was used by the U.S. President, plus countless executives who could afford any device of that era. It was such a specimen of engineering excellence that you can pay in 2024 to retrofit an original Blackberry keyboard onto a modern iPhone, i.e. no one has yet surpassed that physical keyboard.
U.S. tech companies continue to recruit from U of Waterloo.
I guess it's fine that engineers will virtually never move from the US to Canada for better pay and more interesting projects, and always the other way around. It's fine that if you ask a Canadian kid to list off apps or tech products that are cool, they won't be able to think of a single Canadian one (most of them won't even have heard of Bombardier). It's fine that our telecoms are some of the worst and most expensive in the world, that this lack of competitiveness and desire for excellence extends to just about every sector of our economy, and Canadians just roll over and take it. It's fine that the brightest students at our universities will leave for other countries with more opportunities and lower cost of living. It's fine, because remember BlackBerry?
AI mentioned upthread should be relevant to kids. Besides the foundational early work at U of Toronto, there is at least one Canadian AI startup ($3B+ valuation) with global investors and enterprise customers. They have US offices but HQ remains in Canada. Tenstorrent ($1B valuation) silicon startup recruited chip legend Jim Keller (designer of Ryzen & more). Canadian ATI (GPUs) remains part of AMD.
There's a fantastic woodworking tools vendor that's been around for 40 years, earning kudos for both products and company culture, Lee Valley Tools, which sells to both Canada and US customers.
The multi-millennia history of engineering includes the history of military innovation. If Canada is going to be compared to anyone, Finland is a more natural comparison than the southern neighbor with 10X population and 30X military budget.
That's interesting. Growing up in rural Ontario, we didn't know any different at the time, but in hindsight we were way ahead of the curve. For example, I didn't know anyone who didn't have a computer at home by the early-to-mid 80s, even families that were, looking back, quite poor. Apparently that would have been unthinkable for our American counterparts. We had high speed internet to the farm by the year 2000, and in the early 2010s fibre was installed to the farm. That was (and still largely is) unheard of stateside.
Similarly, programming was taught in schools not only during my time, but also my parents' time. They recount writing programs on punch cards in high school. My generation was writing BASIC already in elementary school. Meanwhile, our friends to the south were still desperately trying to get programming into the classrooms just a few years ago!
Suburbs are stereotypically where you find boring people, so perhaps the disinterest in technology that you experienced was simply down to that stereotype playing out? The people around me have always seemed quite excited about technology. Perhaps B.C. was/is behind the curve, but I'm not sure that is telling about the country as a whole. I don't see that being the case at all further east.
> Canada also doesn't have much of an engineering/technology culture.
I posit that Canada's situation is that it is primarily focused on engineering/technology culture at the cost of not being focused on business culture. Canada is quite strong in engineering (and in producing engineers), but as you allude to, since the business culture is lacking the engineering and engineers go to where there is a business culture (i.e. the USA). Engineering for engineering's sake does not stand on its own. You also need business to prop it up. And that's where Canada fails.
Route leaks can happen to anyone, but the fact that it brought down their entire network, including voice and internet services, across all provinces, was unacceptable.
What's even more concerning is that they had no out-of-band access, which meant no management access to their network. This explains why the outage lasted a whopping 24 hours.
In my opinion, the lack of OOB was the most critical and yet the most preventable. Proper OOB is a must; I wouldn't operate a network without it, I don't understand why Rogers thought that was acceptable.
> [...] both the wireless and wireline networks sharing a common IP core network, the scope of the outage was extreme in that it resulted in a catastrophic loss of all services. [...] It is a design choice by [...] Rogers, that seeks to balance cost with performance.
Based on this write up and details I gathered, I believe the root cause, the fundamental reason for the failure is incorrect cost-to-performance balance at senior management.
But yeah, better than. Nothing :)
1) Remote into the boxes to see what's happening.
2) Talk to other devs because their phones are on Rogers network.
My stress couldn't.
Talk about some grade A gaslighting here. Reading the post mortem they first tell you it wasn't a design flaw then say they routed all their data through one core router ( including a lack of a management network). Then they say they are going to fix things by separating out the wireless and wired traffic.
Why would you fix things if it wasn't a design flaw?
Out of band access is like resilient architecture 101. Hell even homelabs generally have some way to do it. It's appalling that Rogers didn't have a way to access the core IP routers out of band. Yes it might mean having to use a competitors infrastructure but they ended up having to do it anyway. And with the failure of the service now all the infrastructure providers are under additional scrutiny. Rogers should be striking some agreements with other providers to carry core traffic in case of an outage such as in this DR situation. For example Visa, MC, Amex all have agreements in place to process each others auth data in case the other party goes down. The thinking here being an outage for credit cards makes everyone look bad.
That said, if 151 Front Street West gets wiped off the face of the planet, the internet is probably going to be degraded for just about everyone in Ontario.