Rogers network outage across Canada hits banks, businesses and consumers
reuters.com
reuters.com
1. A new local player comes to town, builds their own infrastructure (e.g. connects one building to fiber) and starts offering competitive service.
2. Rogers/Shaw/Bell/Telus immediately offer better terms for the residents of that building.
3. The competitor runs out of money and leaves.
4. The telecoms revert to their usual pricing.
This hurts competition, this hurts customers, this hurts long-term infrastructure resilience, but not a single politician ever comes close to even admitting the problem. I understand you cannot do it on the federal level where both major parties are on the telecoms' payroll, but it could be a very low-hanging fruit for someone running for a municipal position to address. But nope, identity politics, words, feelings and fighting climate change with paper straws seem to be what the electorate wants instead. Sad.
It's unfortunate that I need to visit a province that discriminates against my mother with its regressive laws, but at least I can pay my respects and get out without contributing to the oligopoly that runs the country.
Can you elaborate? I only very vaguely follow Quebec news so I'm curious what you're referring to.
The reality is that they were only interested in running those routes as long as another airline was. YKF has been down this road before.
The ride-hailing services pulled this in various places around the world. Airlines in Canada did it as well. Feels like whack-a-mole.
Verizon tried to get in Canadian market in 2019ish but top three telecom formed a mafia and didn't let that happen. Same thing with Delta, would be a much better alternative to Air Canada.
Many counties have their own airline. Canada shouldn't automatically give up our airline in favor of a US one, because the US one wants in.
https://crtc.gc.ca/eng/phone/mobile/occa.htm
Near the bottom it states that those companies are supposed to promote those plans. They didn't the last time I tried looking them up. Regulation is meaningless if there is a lack of enforcement.
(Note: I'm not saying that the pricing scheme is particularly good since it leaves the needs of many people unaddressed. On the other hand, it does address the needs of those who would have the most trouble affording phone service.)
Also consider the continual battle of third-party ISPs to provide affordable Internet access. The CRTC says the major providers have to lease out their lines and stipulate what those rates are. On the other hand, those third-party ISPs are constantly fighting to keep the rates low and are pretty much tied to providing service levels that match the major providers.
For Telus, I had to go to four different plan/pricing pages before I found a link mentioning that cheaper plans are only available on Koodo (their lower priced brand). And then, the $35 plan is as lousy as they can make it and still comply with the CRTC - 3GB of data at 3G (only 2 generations ago, awesome) speeds, with overage costs of $13/100MB. Just absurdly bad. Or maybe you want something more reasonable, let's cut out that data access, you just want talk and text? That'll save you a whopping $3, at $32/month.
So that's the telecom situation in Canada; the first 3GB cost you $3, the next 3GB cost you $390.
So, the various Canadian governments will do whatever they can to maintain corporate status quo and keep the money flowing. That's how it works. Just please big players and the rest follows.
Then, don't expect too much. Canada has never been a real country, its people never decided to take over and govern themselves, it's just an ex-British colonial body from a bygone era (one of many), and has always been acting as such.
Because of that, it's also one of the few places in the world where true personal and community freedom can be achieved, probably because it's so harsh and so vast and so highly dysfunctional and so challenging and so unimportant. Whatever it is, it's nice to be a Canadian for the most part. I'm not proud of being one, but I don't have to be, and that works pretty well for me.
"As it stands now, foreign ownership of a telecommunications company is limited to no more than 20 per cent of a company’s voting shares and no more than 33.3 per cent of the voting shares of a carrier’s holding company, and an effective total limit of 46.7 per cent (as long as the foreign entity doesn’t have control). On top of that, at least 80 per cent of the board members must be Canadian citizens." [0]
(Although seems not to apply to small actors under 10 percent of the market share anymore)
[0] https://financialpost.com/telecom/tight-reins-leaves-our-tel...
The industry is large/stable enough to allow 100% foreign ownership.
Price transparency in general is a huge issue with Canadian telecoms, it's well known that it's a "complaint based system" with the big three. They'll hook you with an introductory rate and then start jacking up the prices after a year or two. The usual dance is you go to one of the other big telecoms and get a quote for their introductory rate, and then go back to yours and threaten to leave unless it's matched. It's a pretty awful system that requires people to know how to navigate it. Most don't, so after their intro period expires they just get gouged. My father in law is paying almost $200 a month for internet and TV.
I was about to write a whole comment about how I've never seen this myself and to disagree with you I was going to compare the price of Bell Fibre in my building to Beanfield. The latter of which I have at $50/mo at 1Gbps.
But then I loaded up Bell's availability page and entered my address... it was almost like a sick joke. The spinner was saying "Checking availability" or something like that. However, I could see the page with prices behind it darkened out while it was running. It was $115/mo for 2 years and $125/mo after that for 1Gbps fibre. But when the spinner was done and they "checked my availability"... well what do ya know! It's now magically $50/mo. Pretty disgusting that they can get away with that.
A better way, and likely the thing in play here, is to price according to a mix of value provided, and the ability of the market to pay.
If you are spread enough this will result in some markets paying effectively a large markup, and done a much smaller markup, and some, potentially, less than cost.
The reason the package costs $300 is because that is what the market will bear, and the utility to the customer exceeds the value of that cash.
Incidentally that value may also be in support etc.
In summary, markup is one of the least effective pricing methodologies, and invariably leaves a lot of money on the table.
Competition is not the only thing, in France we have 4 highly regulated fiber providers who must compete or pay fines, but in HK it's way more shady and we choose building and flats to get the best ones (cable is shit, former public telco is expensive, new kid is fast and cheap). It's still nowhere near your prices: a shit cable is 98HKD and an expensive HK Telecom is 500...
It’s cartel behavior at its best. The Canadian telecom landscape is a sick joke.
The same argument is used for keeping the postal service public when you could just subsidize the parts of the market that wouldn’t be self-sustaining rather than propping up the entire money-losing crown business. Burning tax dollars at the parcel delivery business in the age of e-commerce is a sad joke.
The whole “rural people are being protected” seems like a cover story for powerful friends of politicians to keep the money train rolling, and regular people eat it up on social media.
Plus Starlink et al has rendered the rural argument moot.
That's still pretty good. In Austria that's 80 euros since telecoms here have the same government protected cartel as in Canada. And it's not even available in every building of the city, but you need to check before you move whether you have fiber or not in the building.
Similar in Germany.
One of the things I miss from Eastern Europe is the fast and cheap internet that's available in any building in the city.
ie, another player shows up in the market, offers a more attractive price and forces the other players to reduce their prices or shed customers.
It's not like Bell, in this case, were offering 1Gbps for something absurdly below cost, like $1/mo or something just to drive the new comer out of business.
Matching and lowering prices is good for everyone. The new player running out of money and going out of business is really just a lack of foresight and planning on their part. It's pretty absurd to not expect the incumbents to reduce pricing when they are directly completed against.
But they should be forced to reduce it everywhere where they're offering similar service, not just a limited local area.
Excpet they revert back to old prices once the competition is dead. This is textbook predatory pricing.
> It's not like Bell, in this case, were offering 1Gbps for something absurdly below cost, like $1/mo or something just to drive the new comer out of business.
What's the significance of "absurdly below cost" here? They are doing exactly what you describe except the exact number here is not 1 but 50.
This is because if a large enough company does this, they can lower their prices locally to below cost whenever a new company enters a local market and subsidize this with their other markets. This creates a stranglehold on the market that can allow a company to charge artificially high prices for sub-par services.
This isn't just theory there is a well established pattern here and that is why laws prohibit it in many places.
No, because any time a company shows up that could pose an actual challenge to their stranglehold, all 3 companies gang up and lobby against them.
E.g.
https://globalnews.ca/news/753408/new-ad-against-verizon-exp...
https://www.iphoneincanada.ca/carriers/rogers-telus-bell-job...
> Is this not just competition working the way it's supposed to?
Yes. This is competition working exactly as you would expect it to in a free market: One company is a dominant supplier, and they are able to match rates for a new entrant. They are able to out compete this new company, and they will do so - more marketing, lobbying, cost matching (or undercutting), etc. They have the ability to beat the competition, and they will.
So everything is going exactly as you'd expect - except the points from the parents comment still apply: the consumers in this market are actually getting fucked - the price drop will be temporary, the competitor will be forced out of business and leave, and the total infrastructure investment in the area will go down.
This is what's termed "Market failure". The free market here operates in a way that doesn't increase the well being of all (or even most) participants.
So yes - this is just competition playing out as we'd expect in a free market, but instead of doing what it normally does in an area (force infrastructure updates, service improvements, cost reductions, and higher efficiency as companies compete - all good things) it's doing bad things. Why?
Well - this case is the literal textbook definition of a "Natural monopoly". It turns out that when competition appears, Bell is able to outcompete them not by actually improving, but by leveraging existing infrastructure and scale in a way that the startup company cannot.
The free market isn't making Bell better - they're not having to work any harder or improve. That's great for Bell, but pretty bad for everybody else.
So (at least in theory) we regulate this case of the free market, because we've seen that "normal competition" doesn't actually work here.
Recently our telecom regulator (the CRTC) was considering whether MVNOs should be allowed in Canada. Our Competition Beaureau, which is supposed to be enforcing antitrust laws, provided guidance to the CRTC[1]. Here's an excerpt:
"MVNOs can drive lower prices and greater choice, but they also could threaten the demonstrated progress in enhancing competition in this industry to date."
That's right. "Increased competition and lower prices is bad for competition." "Demonstrated progress" in a country with some of the highest telecom prices in the world.
Let's not even get into how the chairperson of the CRTC is literally a former telecom lobbyist.
Ultimately, MVNOs were "approved", but with practically insurmountable requirements for new entrants, effectively blocking them.
[1] https://www.competitionbureau.gc.ca/eic/site/cb-bc.nsf/eng/0...
Which would have the dual purpose of employing researchers and public outreach and have powers to stop hires, limit grants/subsidies, and collaborates with with anti-trust.
But who are we kidding, politicians never bite the hands that feed them.
My only notions so far:
- Add explicit carve outs for customers (of the regulated entity). Where the representatives are chosen by jury duty or sortition or other suitably random method, so they too are resistant to capture.
- The customer's have veto power. Maybe thru an ombudsman structure. Maybe thru majority of the regulatory body.
- Like citizen's assemblies, customers have investigatory powers. They can ask questions, they cannot be denied. They benefit from expert testimony.
- No artificial deadlines for deliberations. Some decisions just take longer.
- All decisions (laws, rules, procedures) have built-in expiration (TTL), and must therefore be regularly reauthorized.
- All artifacts and hearings made public by default.
- Pay board members and their staff. So that normal people can afford to serve.
> a big investment in an anti-regulatory capture organization
I think you're on to something.
I keep thinking of the law practice that makes terrific money suing pharmacy benefit managers (or maybe just the pharmas themselves) for violating pricing rules. Sorry, no cite.
Oh my sweet summer child.
The Canadians learned this scam from us. Centurylink has been doing this for decades. They ran several WISPs out of business in Bellingham, WA.
> 1. A new local player comes to town, builds their own infrastructure (e.g. connects one building to fiber) and starts offering competitive service.
You forgot to mention that whoever is the ILEC in an area (Telus, Bell, etc) and/or the incumbent 35+ year early advantage local cable TV operator (Shaw, Rogers) has an immense advantage in owning and controlling existing right-of-way to reach whatever is the last mile service delivery location, whether it's aerial pole to pole or duct routes.
of course in places where the local phone company or cable tv company shares aerial utility pole based infrastructure with the local power company, they're very close buddies as well
The recent "RDOF" mega-grant has one of these clauses (at the census block level):
"All support recipients must serve locations newly built after the revised location total but before the end of year eight upon reasonable request"
https://www.fcc.gov/auction/904/factsheet
These mega-grant things happen every 6-8 years. The last was "CAP II", preceded by "CAP I", etc. The telcos are basically treated like municipalities when it comes to federal grantmaking. This is basically the system that was lobbied into place after AT&T was broken up; the FCC just took over AT&T's local-loop capital allocation.
Such behavior spawned lots of laws designed to prevent it from happening again. And, just like the telecoms extort huge subscriptions from their clients using extreme product bundling, politicians engaged in ideology and law bundling to get voters to flock to them attracted by some laws, enabling them to enact (or in this case, repel) the laws preventing them from going back to 19th century capitalist practices.
Not that I find it a particularly good design, though.
> was designed
We can't have a hundred different companies burying infrastructure or stringing up lines on poles.
This weak apologia for monopolies is what keeps them around. Even when the cost created via gov-backed pseudo-private monopolies are obviously worse than their hypotheticals.
A security-emitting entity doesnt need to pay back the debt too much (it should but doesnt have to, via dividend) and therefore can afford to try things to raise returns for both shareholders and management and absorb failure.
The worst situation is when a state utility tries to innovate and fails: this leads to privatization which actually transforms it in a public companies (the words are weird: state companies are not directly publicly owned by the citizens, private companies on the public market can be). If a public market private company fails to innovate and just produces riskless cashflow forever, it becomes a good candidate for state private ownership.
One cable/fiber, rented by all providers. The network only has one goal: distribution. The providers are the ones competing.
https://foxnomad.com/2012/03/15/why-is-the-internet-in-roman...
aka Pay to play.
That means "I" (my building) could build my (our) own fibre connection and then use that as leverage to get cheap internet without having to actually run it. It'd be a bit silly, but sounds long-term profitable.
The focus should be on enabling this sort of competition.
https://www.theglobeandmail.com/news/national/john-a-tory-a-...
I think the best way to address it is to open the industry for disruption by encouraging new players(without Billion dollars in market cap) by removing barriers for entry, With stringent monitoring/punishments for anti-competitive behavior including backroom deals among carriers and mobile ecosystem duopoly.
we can all agree politicians are supposed to represent the population of the country not the company and their board of directors.
They have no presence outside the country - they are so bloated and inefficient they couldn't compete - but they are omnipresent here.
There are lots of people who depend on Rogers for home Internet, home phone, mobile phone, TV, and home security, not to mention business services that consumers also depend on, like the payment networks that are down.
Plus their media properties, which probably come in useful when the government starts to think about allowing more competition or decreasing the taxpayer money being funneled to them.
In my neighborhood, I have one effective ISP, that is the cable company (who offers phone+internet+cable), the other is an old-school phone company who offers DSL (with bonded DSL, up to 12mbit). I thought that I could get a slow DSL connection to back up my cable, but no, there's some neighborhood line concentrator that means I can't get DSL at any speed.
So I really have only one ISP to "choose" from. (well, in theory I could get Starlink, but my area is still waitlisted, as is Verizon 5G home internet)
Unlimited (international!) roaming and everything isn't that expensive (Only about 100$) compared to whatever insane amount I'd be paying if I had some ripoff, under 5 gigabyte or whatever pittance data capped Canadian phone plan for no reason. I also maintain a Central American phone plan that doesn't cost me very much either (2$ every 3 months just to keep the number outside of the country and receive SMS/use chat apps basically, but I don't do roaming when out of that region - When in the region, I spend maybe 20$ a month).
Since the Rogers website is down (Great sales strategy!) I can refer anyone curious to a portion of their menu from some image I found; [1] 95$CAD (500MB until you hit overage) + 60$ for 'roam like home' (no mention of data, canadians still use SMS way too widely), and then probably more to up that data cap with no guarantee that applies to roaming? Yeah, no thanks! Oh yeah, the cops just install malware on your shit with reckless abandon, too. [2] I see no reason to consider Canadian nationality as anything but a nice passport at this point.
[1] https://cdn.mobilesyrup.com/wp-content/uploads/2018/05/roger...
[2] https://www.theguardian.com/world/2022/jul/07/canada-police-...
Like what are you even paying roaming for if they pull stuff like that? Is there any acknowledgement that some of us are actually abroad most of the time?
That they can get away with this false advertising astounds me to this day.
I do a lot of my work off of a mobile hotspot so I'm not tied to an office or home. I try to make the most of what's paid for and it goes very far.
I think that if you're getting a decent price it's probably just customer retention at that price point. If they're ripping anyone off widely at the prices I saw, it's new customers... Immigrants etc, pretty nasty.
Same for banks in Canada. It’s unbelievable what they do and get away with.
And airlines. Rather, Airline.
I believe there’s a cultural element here: Canadians are more risk adverse and place a higher value on “established” companies and brands. This influences consumer behavior and legislation (Canadians don’t seem to care about laws that make it hard or impossible for market entrants).
I think it has to do with the fact that historically much of Canada's lands/provinces were literally just land owned by very massive British crown corporations. Massive corporations are baked into the history of Canada as a political entity.
There is no such thing as a "British Crown Corporation."
Crown Corporations are a Canadian thing.
I lived in Canada for 30 years and now live in the Scotland. CalMac and Scottish Water are not "Crown Corporations," but are nevertheless owned and controlled by the Scottish Ministers. This is contrast to BC Ferries, ICBC, and BC Hydro which are Crown Corporations.
In the UK there is something called the Crown Estate, which is somethint again different.
Crown Land is land owned and administered by the provinces. Crown Land can be licenced for many uses and sometimes it can be purchased.
Also, Provincial Parks are not Crown Land.
Canadians see the Canadian Royal Family as nothing more than a figurehead of the state.
I am a Canadian and I have studied British and North American History extensively.
If you talked about "crown corporations", or "Regina vs." (for criminal cases) no one would associate that with Elizabeth II. "Crown" and anything like that basically just means government.
HBC lands were surrendered to the British government in 1868 ahead of confederation.
Crown land is owned by the Canadian Crown; the monarchy owns all crown land officially. It's administered by a split across federal and provincial jurisdictions.
(From what i understand. IANAL)
No, Crown Land is controlled by the provinces. No, the Canadian monarchy does not own crown land. The Crown owns crown land. Essentially The Crown in Canada is its own legal entity and instrument for the purposes of administration of public lands. The Crown Estates of England and Scotland are something different again, and the Crown Estate owns things like tidal lands and actual real property.
There is Federal Crown Land, but that is almost completely in the northern territories. This is down to Canada's constitutional makeup because provinces control resources.
The Federal government can expropriate land from the provinces when it has a reason to do so like for military bases.
Here is a link that shows Canada Lands. The white area are Provincial Lands.
https://clss.nrcan-rncan.gc.ca/data-donnees/sgb-maps-dag-car...
Resource administration is a constitutional power that provinces administrator.
The area in white in the below map are not Canada Lands which makes them Provincial Lands, ie Provincial Crown Lands.
https://clss.nrcan-rncan.gc.ca/data-donnees/sgb-maps-dag-car...
TD has more US than Canadian branches.
But Rogers, Bell, Telus? Nobody has heard of them ex-Canada unless they’re an ex-resident with bad memories.
But it’s also true that over the years, TD has had to merge systems and make changes. They’ve been operating in the US for about two decades now. So they’ve had time to merge and continue growing by merging companies in the US.
It’s true though. They aren’t entirely integrated across borders the way other companies might be. The plans and cards they offer in the US are a much better value than those in Canada, too.
At least a few years ago that wasn't possible.
Canadian banks are so bad that they actually make the service and fees with a USD checking account with all the normal features at Wells Fargo look good.
It's probably the density and income levels that make it worth putting out the infrastructure
Now I have little doubt that if foreign providers were allowed in Canada that Verizon or AT&T would be able to setup reasonable competition in short order.
I expected more uptake since it would have been a better deal for most Canadians in those urban centers, but I think there's a subtle bias against taking these kinds of risks.
I think had they focused on southern Ontario or BC and expanded from there rather than aiming to hit the all the big metros first they might have done better.
IIRC Wind was offering unlimited plans at 50% the price of a mediocre plan on major providers. You could have hours of roaming calls and still come out ahead (and this is in the edge case, where you happen to go camping that month). Most things being equal, Canadians broadly prefer to pay more to have a sure thing. It’s not a bad instinct, but it has consequences to how industry is shaped in the long term.
I would dispute this one, only because in addition to Air Canada (fleet size 312), Westjet does exist (fleet size 162). And the other regional airlines offer (-ed, prepandemic) some price/service competition. Comparisons to the US aren't as useful with airlines as with some other industries due to population differences.
Edit: and the union Pearson Express made the main airport a lot more accessible
Not ... exactly. But effectively, yes.
A combination of a tiny population (5.1 millions) and a long way from nowhere (roughly 3,200 km / 2,000 mi from Sydney).
Australia and Canada already have issues from their own small scale (and vast land areas). NZ takes it up a notch.
I dropped TV and a landline a long time ago, and use cell (google-fi/t-mobile).
The annoying thing is all the spam mail I get from spectrum to bundle all the above for much lower than I'm paying now for an introductory rate. I don't want to play that game but it kills me what I could be saving for at least 6 months to a year.
"Canada is an endangered species protection wildlife reserve park for dinosaur telecoms"
It's a travesty that the government is actually going to let Shaw and Rogers merge to even further concentrate power in the hands of a few families and reduce market competition.
The regulatory environment Canadian telcos operate in is completely captured.
Google have tried to enter the FTTH market thinking there must be opportunities to innovate and reduce the cost to consumers. But even they have struggled due to the complexities of deploying fiber in the existing built environment...
Is there any place in the world that can claim to have a diverse, competitive Telco market?
a) last mile dark fiber network operated by local public utility district which is also the electrical grid operator, and rents access to the fiber to 3rd party ISPs
b) local cable tv operator, legacy coax operator, often docsis3 cablemodem
c) local ILEC/POTS phone company that may or may not have overbuilt its last mile copper/DSL service with its own singlemode fiber and 1Gbps GPON service.
for primarily mobile phone carriers like rogers, there's an effective RF planning limit of around four major LTE/3GPP technology based operators in any given geographical area.
USA used to have 4 with sprint until the tmobile/sprint acquisition.
and its not clear how individuals are being handed out rights to say run a casino in canada.
if you took a look at canada's corruption, you will be shocked. we don't quite live in a first world even though we like to tell ourselves all of the government officials/servants are honest.
after all the canadian embassy staff in Hong Kong happily handed out PR residency to hardcore organized criminal groups in exchange for various luxuries and perks. when a staff tried to expose it, he was quickly removed and media began to attack him.
its not only that but you see NGO's championing for racist white supremacy linked groups who vandalize Chinatown in Vancouver and with a large chunk of locals who feel that they are "being overrun by a certain ethnic group" eat that up and the same populist individuals get elected again and again. then my tax goes towards those interest while none for me because i'm 'privileged'
in the long run, I see this system breaking down, if not already. Canada no longer feels like a country but some feudalistic interest group driven, poorly run corportation.
so glad i dont have to pay taxes here. the savings and currency difference allowed me to create jobs in another part of the world. there was a time where I hoped things would get better and I could be creating jobs locally.
i just regret wasting my youth in canada and west coast. canada is a bubble and lot of us are moving capital/jobs out of it.
why contribute to a country that just sees you as an ATM to transfer payments to others who blame everyone but themselves and constantly wanting hand outs?
im done with canada and im warning anybody who still attach romantic outlooks, especially in heavily marketed cities like vancouver.
Or you could put the fox in charge of the hen house and end up in Australia's situation, where local monopolist Telstra owns the phone network and is supposed to play nicely with its competitors.
There are 5 different mobile providers in Singapore: Singtel, Starhub, M1, MyRepublic, ViewQwest and WhizComms.
Many of them also do fiber broadband. Some government body owns the infrastructure and the providers provide the connection to the internet. There are 5 providers as well.
I was paying $50/month for 1GB fiber and I paid more since I did a shorter term plan. If you buy in for 2 years, it's can be $35/month.
And this is in a country of 5.5M people.
BT/Openreach has control of about 80% of the backhaul/trunk connectivity.
Hence you have a very competitive market landscape in the UK. There are at least 5 major national retail players using various wholesale products (BT, VM, TalkTalk, Sky, Vodafone), plus dozens of altnets offering FTTH on totally seperate fibre infrastructure have started (Hyperoptic, Cityfibre, GNetworks, Community Fibre).
In my flat in London I have access to 4 seperate FTTH networks (with completely different infrastructure) - Openreach FTTH, VM DOCSIS, Hyperoptic FTTB and Community Fibre FTTH. The market works here, prices are low and there are very few data caps.
Yip, things in London are quite different than the rest of the UK.
While Openreach has to allow access to other providers to its infrastructure, most infrastructure is owned and controlled by BT.
If you want FTTP, the underlying service is still BT Openreach. Openreach is very profitable.
At my previous employer it cost us £30K to have BT put in 3000m of fibre that took them 1 afternoon, then it cost £1K a month for a 30Mbs service.
BT owns provides the vast majority of broadband service in the UK and it is somewhat disingenuous to claim BT is on an equal footing as Hyperoptic, VM, or Community Fibre.
TT and Sky use BT OR infra, and EE, Plusnet, and BT Retail are all owned by the BT Group.
The telecom market in the UK is very consolidated and controlled by BT.
If you have a new-build flat in a major UK city you probably have a similar decent choice but the rest of the housing stock (the majority by far) is stuck with BT/Virgin.
VM can be ok or can be congested depending on the area. BT FTTH doesn't suffer from congestion issues (nor does the FTTC).
Check https://bidb.uk/ for more information. It collates all the altnets and planned roadworks into one dashboard.
That is just not true. BT consumer FTTC is contended up to 40 to 1--that is you are sharing your backhaul with upto 40 of your neighbours. I am assuming that their FTTH services are the same.
Not always. There are many altnets now (cityfibre etc).
> TT and Sky use BT OR infra, and EE, Plusnet, and BT Retail are all owned by the BT Group.
TalkTalk also use cityfibre.
> The telecom market in the UK is very consolidated and controlled by BT.
If you mean physical infrastrure, yes it is (though changing rapidly). Consumer level pricing is competitive though and the services are reliable in the large. It is a different planet compared to the US and especially Canada.
To me it would be shocking if they were providing updates. Canada’s telecommunication monopolies are not known for their customer service and, even worse, they seem to have internal corporate cultures of entitlement and arrogance that drive an “F U” attitude in general when it comes to external communications and accountability. So you won’t see a post mortem. You won’t see timely updates. You certainly will never see some kind of status page!
I’m sure Rogers will be forced to do a post-mortem for large commercial clients where contractually required, but I highly doubt we will hear a word from them about the cause of this or what was/is being done to rectify it. They probably have an ETA but feel no obligation to share it.
Rogers isn’t alone. I would expect the same behaviour from Bell or Telus. Canada has serious issues when it comes to its big telco carriers and it represents major risk to the Canadian economy.
I actually think this outage will show that there's no major risk to the economy and every Canadian can just take every Friday off with no ill effects.
It's not just Rogers' client who are impacted, the whole banking ecosystem is impacted.
Even rumors that Costco can only accept cash, the Capital One Mastercard network might be down.
I don't see how Rogers could keep quiet on this, unless it's from some malicious actor.
My sweet summer child, you have no seen how meek and docile Canadians are.
They'll take a page from the Federal Liberals, spout a bunch of unrelated platitudes and nothingness, and pretend nothing happened.
The oligarchs control Canada and the Federal Liberals are just a finger in their hand. There will be no inquiry from regulators - what's more - they'll rubber stamp the Rogers - Shaw merger that's been in the works.
If Canadians say anything, Bill C-11 will be ready to squelch them online. If they say anything in person, a quick speech from Trudeau calling them foreign-funded MAGA terrorists will cause the government-funded CBC to produce an article. That article will then be used by the Trudeau Liberals to invoke the Emergency Act once more, then de-bank and trample them.
https://www.cbc.ca/news/canada/calgary/coutts-protest-blocka...
It is odd they don't have PR people trying to spin it. But I am not surprised in the slightest that there is no technically useful updates happening.
Canada will get the same action on this as Americans do about mass shootings. “Thoughts and Prayers”.
The shocking part is not that they are giving no updates. Everyone is to expect that. The shocking part is that we (collectively) don't care that they give no updates, and let them get away with it.
This outage is way too disruptive for polite exchanges of platitudes.
The public have to make their point of view high-key known. The public should respond so strongly that the other companies start saying things in public. Then the rich will get queasy and ask themselves, what do we have to do to buy back social peace?
Vote with our wallet? C'mon, what else?
>What actionable steps can consumers perform?
I'm sure there'll be industry wide participation in a post-mortem. Translation for non-Canadians... The head of the CRTC will book a tee time for him plus the CEOs of Rogers, Bell, and Telus.
Trust an American company to come up with a clearer picture of what's going on, highlighting the mediocrity of Canadian businesses.
A lot of us are in IT and for a network of national importance in a "developed nation" to be taken offline so easily is worrisome.
- Point of sale machines. Supermarkets are accepting only cash.
- ATMs. Several banks are incapable of dispensing cash.
- Public transit ticketing systems. The TTC is effectively running for free, today.
- Public bicycle rental stations.
- Public parking locations.
How long before the public realizes that we are not Rogers' customers, but its hostages?
> ATMs. Several banks are incapable of dispensing cash
Back to bartering.
1) the site has connectivity so only Interac is down
2) the site has no connectivity and can process credit transactions offline and queue them (don’t tell anyone their expired card will work), but not for debit.
The expiration date is stored on the card, so expired cards wouldn't work. Revoked cards (eg. ones replaced by the bank) would, though.
Keep in mind that the merchant sets terminal limits for what transactions are allowed to be processed offline, so one might be able to get a burger without connectivity, but not a new TV.
The Ebox and Teksavvy are great, but like you say it's hard to survive on other people's infra, that's why big cities need to make their own.
Won't happen. The average Canadians are one of the most sheep like docile people out there.
That happened because of a hostile attack ad campaign by the big 3.
Yes, I'm being pessimistic, but it's the way things are.
This video shows a senior person at Rogers giving an update on the situation. It feels like I am in the twilight zone: https://www.youtube.com/watch?v=MYfRZZaPrC0
BBC: https://www.bbc.com/news/world-us-canada-62102223
NY Times: https://www.nytimes.com/2022/07/08/world/americas/rogers-int...
Reuters (doesn't really count, because HQ is in Toronto): https://www.reuters.com/business/media-telecom/rogers-commun...
NPR: https://www.npr.org/2022/07/08/1110500480/rogers-canada-majo...
Bloomberg: https://www.bloomberg.com/news/articles/2022-07-08/banks-pay...
(Luckily my phone is Freedom and my Internet is an independent small Hamilton local ISP.)
For me, I'm rural and I use a 900mhz point to point wireless connection that caps out around 15-20mbps on a good day. The upside is it's provided by a local old school indie ISP ("NetAccess"). It's $130 CAD a month, which is... expensive... but unlimited which is far cry from the theft that Rogers was performing on me prior with the 3g (then LTE) connection I had that at first capped me at 25G per month @ $100 a month and eventually went up to a "generous" 100G and then charged insane overage fees. My teen daughter ran us up a $400 bill one month.
Bell and Rogers are now offering more competitive rural options (at the behest of the gov't) but I refuse to give them any more money.
And Rogers broadband is mostly just Ontario?
re:landline, I'd say go a step further, and don't use cell services at all. All my phone calls use Signal or VoIP, sms is routed through VoIP.ms. My cell is data only and costs 15$/month for 3GB (fido/rogers).
I know, I know, some conditions may apply, but I really hate telcos and enjoy finding workarounds. Hopefully this will encourage a discussion about monopolies/redundancy.
> redundancy
You do have a point in general. I wouldn’t put VoIP.ms and redundancy in the same sentence though. Unless they’ve drastically changed their infra, they had a massive 15 days (yes) outage not that long ago because of no redundancy and a poorly architected infra.
In the ISP space, there is a bit more competition. Namely, Shaw provides additional coverage in some regions of the country. However, Rogers wants to buy Shaw. You can imagine how bad that will be for Canadians.
I do wonder what the Rogers outage is about. Ransomeware? State attack? Something stupid? If anything, it shows how we should not have critical infrastructure centralized. Competition between ISPs is important.
In the western provinces, AB and BC, Rogers runs a very widespread and strong LTE network. What they do not have is a DOCSIS3/coax and GPON cable TV plant. Nor do they have much terrestrial right of way for aerial plant or underground in conduits, which is where Telus (the historical copper POTS ILEC) and Shaw (the historical cable TV operator going back to the mid 1970s) are by far the strongest.
Rogers is a facilities based last mile cable operator/terrestrial operator in Ontario.
Shaw runs the landline cable tv networks in most of the metro Vancouver area. And many other small to mid sized cities in the west.
Letting Rogers control both one of the largest/strongest mobile phone networks and the only viable land line terrestrial broadband competitor to Telus by acquiring Shaw's cable tv plant is an absolute outrage.
To use a USA analogy, it's like if T-Mobile already owned RCN (Astound) in some big part of the country and then proceeded to buy Comcast.
Or if Verizon bought Spectrum (Charter/historical TWTC cable).
Whatever it is… it's definitely something stupid.
They announced 216.176.216.0/21.
Interac's backup network also happens to be with ... Rogers.
Canadians across the country can't use debit cards, regardless of their financial institution. Credit and cash only. E-transfers are unavailable. Fun for people trying to pay their bills online.
And that's just the financial side of the situation. Millions of people have lost Internet and phone service.
Someone on Reddit wrote: "Maybe the worst part is that they have literally no open lines of communication. No twitter posts. All chat options (twitter, IG, FB, web based) are all unresponsive. Can’t call tech support, as the call will fail. Can’t even call billing or general inquires as it’s the same thing. Shit happens, I get it. But to leave your customers completely in the dark? Brutal. Just brutal."
Also, around half my team are unable to effectively work today. We all work remote, and half the team's home internet is on Rogers, which is down. Some people are tethering, but people who are also on Rogers for their wireless are out of luck.
I'm not a Rogers customer anymore, but my boss showed me how you can't even login to their account anymore. Their website is down too, just timing out.
Funny, I left Rogers last year and just last night a telemarketer phoned me asking me to return to Rogers. What a sign that I made the right call by politely refusing.
It seems like Bitcoin and crypto fared well during this outage.
So no online banking, withdrawals, payments in affected parts Canada.
And a society that refuses to invest in robust infrastructure.
I really hope this results in the CRTC starting to allow more competition. I hope, but I'm not optimistic.
It's quite astonishing by any reasonable standard, but this is Rogers we're talking about, so pretty much par for the course.
it seems like the BGP traffic signature of any isp-level outage, regardless of the cause, would look like this. they went down, so they stopped advertising that they could accept traffic. then they tried to accept a little bit of traffic, but that didn't work so they turned it off again.
It feels like there's a big, fundamental problem affecting a variety of Rogers infra, where BGP and cell problems are just symptoms we can observe. CloudFlare says it looks more like a failure than an attack; I'm not entirely convinced.
I’m in Canada at the moment with my UK SIM and using Rogers internet just fine. Completely unaware of any problems until my friends warned me about needing cash and I then double checked and still, phone internet via Rogers on a roaming SIM worked fine.
My guess is their voice, sms etc all are IP these days which was impacted by their BGP issues. Whereas my UK carrier roaming gives me a UK IP still, unrelated to Rogers advertised blocks.
Impacted, yes. But "no service"? That's dubious.
I hear you on foreign SIM working. I got online at a cafe just long enough to buy a Yessim eSIM. When I activated it and turned on data roaming, I had LTE working on... the Rogers network! That I didn't expect.
Nope, in contrary I will fully believe it. Circuit-switched networks are dead, period. Most "circuit-switched" networks are actually IP networks just with custom FEC'd and prioritised protocols so that there's backwards compatibility. Most Subscriber Authentication systems since UMTS and 3G CDMA (except for super-early deployments, and since they have LTE they would have migrated it if it were the case) have moved into an IP-based system because it's cheaper, and if that's down then everything is down.
https://en.wikipedia.org/wiki/July_2022_Rogers_Communication...
Using Fido for mobile internet access (also by Rogers).
Zero internet. It's wild to discover how reliant I am on the internet:
- Can't get around this city (no waze, gmaps).
- Can't figure out what's going on (AM radio is non stop commercials)
- Can't figure out where we might get some internet (maybe WeWork somewhere? How do you find one without internet)
This is just wild. Almost as critical as no water, or no power in the winter.
I expect the device I send this from will be offline again in the next few minutes, but I’m willing to be pleasantly surprised.
Fun stuff!
offline gmaps!
Thankfully Canada’s horrendous roaming and overage charges has taught me how to use my cell phone effectively without service.
(Also have the entirety of Wikipedia on my phone, thanks kiwix!)
https://en.wikipedia.org/wiki/Kiwix
~88 GB w/ images, English. 47 GB w/o.
There are also Wikipedia apps which opportunistically download pages. You'll have access to what you've previously read offline, but will require connectivity to read more.
Don't you have public service? Isn't that what CBC Radio One is, with no commercials?
680 news is owned by rogers and cp24 is owned by bell lol!
"we're getting very close to understanding the root cause of the of the failure. And we're taking actions along with our network vendors to recover the situation."
"We don't understand how the different levels of redundancy that we build across the network coast to coast have not worked," he said.
This network has definitely rogered itself.
Mobile and data service has become quite embedded in daily life.
Unfortunately, I've been using that number for most of my 2FA so essentially, I've been unable to access a lot of services because of this (ones where I didn't use Authy/Google Authenticator).
Lesson learnt: use a separate VoIP business phone number instead.
At least, I'd be getting verification codes without having to worry about the service being down.
Really strange day but not unexpected. We “know” the telcos are screwing us but we don’t have a way to gang up against them, as people easily buy into their feelgood propaganda.
In terms of causes, I think it was one of a few possibilities in order of percieved plausibility and likelihood:
1. newb engineer updated router configs using a CI pipeline that caused them to stop talking to each other.
2. standard mistake of redistributing iBGP with a bunch of static routes for load balancing into eBGP causing their internet facing ASN to announce the internet, and it took down their internal MPLS routes for backhauling their cellular traffic over IP.
3. geopolitical situation where someone was making a deniable example of their capability against a target who can't fight back. That was just a friday in Canada, it's just as easy to do D.C., Chicago, or New York.
4. a political stunt and pretext for "a safer internet" with "more oversight," orchestrated by party hacks, or an exercise to see how effectively they could do this. Least likely, but that's the level of trust there is in this govt right now. If you don't want conspiracy theories, try being credible enough people don't have to invent explanations because nobody believes you are in control.
I'd bet on 2 or 3. Items 1 and 4 are weak ideas, but 4 will likely exacerbate confirmation bias. What will likely not come out of this is liberalization of the telecoms market to add competition. However, regardless of teleological fallacies, I don't see this as a crisis that will be left to go to waste.
It has been a good fire drill for business continuity and preparedness anyway.
Seems as good as any time to share a Rogers story with HN:
My mum passed away in 2009, she owed Rogers $200 and change. We took over her contacts/mail forwarding/etc to tie everything up. A month or two afterwards; we got a call from Rogers looking for their money, explained she passed away, they said how sorry they were and figured that was the end of it.
3/4 months after that we got a call from a collections agency, mentioned her passing, got apologies and hang up. This then continued every few months bouncing between Rogers and a collections agency. For some reason they kept passing the debt back and forth? The frequency was low enough, that we never bothered going any further with it, after a couple years it was less annoying and more a nice reminder of mum :)
It took about 6-7 years before they finally clued in, and that's why I'll never have anything to do with Rogers.
it's hard to build resilient rings of fiber between towns/cities where there is only one linear path (road) for right-of-way that you could economically build the fiber along...
For wireless Freedom Mobile was independent from 2008 to 2016. Mobilicity was independent from 2010 to 2015.
They were always cheaper than the incumbents. However, they struggled to get customers because their coverage and speed lagged and they were eventually bought by incumbents for their wireless spectrum license.
The government tried to foster competition by letting them bid on spectrum set aside for new players. And removing strict foreign ownership rules.
Canada is a small market. Geographically larger than the US with the same population of California.
It's just not an attractive telecom investment.
[1] https://twitter.com/HNStatus
[2] https://www.reuters.com/business/media-telecom/rogers-commun...
would love to see a tech write up or blog post or even a Twitter hint but there seems to be absolutely zero transparency.
Closer to 12 hours now I think. Certainly it was out at 7am EST this morning when I woke up.
It would be beneficial to have unvarnished tweets from someone in a technical role, written for technical peers rather than the general public. The press and tech journalists can filter the jargon into something useful for customers. "Dashboards are showing ABC, we're trying XYZ, so far it's looking like LMN", even if the public don't understand what is being said, is more reassuring than a generic statement. It suggests competence and hints that yes, it really is a difficult problem.
Maybe when it destroys the whole Canadian economy for a day
Won't happen. The average Canadians are one of the most sheep like, docile people out there.
Glad that Best Buy has a large stock of Telus SIMs.
Never again, seriously.
This outage was eye-opening. You’d think that Interac (a prolific payment method for using your bank account to pay for products that hit the scene early enough to get entrenched in Canadian habits) would be infrastructure-agnostic, but it’s clear that they are extremely dependent on Rogers’ network. All Interac was down. Places still processed credits cards as normal, the machines still worked, and it didn’t matter what ISP was used by any particular store.
I do not understand how they were comfortable floating billions in transactions over a network with a single choke point.
Rogers is one of those networks, and it basically covers 30-40% of the country's communications.
There are a few smaller regional players, such as Shaw, Cogeco, Videotron etc. I think we might be the only developed country that's allowed this. I know in the US, there are 5 major mobile networks, some of who offer internet etc., but there are major internet providers separate from the mobile network providers.
The reason this has happened, is because building networks is really expensive making it so that being a regional provider is a tough business, and the 3 major companies have played the laws to their advantage. They've done acquisitions over time, and used the rules around foreign investing to block a lot of major development in 3rd and 4th networks from happening.
This is why the Rogers/Shaw deal should be blocked and the government needs to allow foreign investment to be able to come in and build out networks.
This problem can only be solved in 1 of 2 ways, and only through regulation unfortunately: - A complete overhaul of our rules which will allow foreign investment in our communications networks in Canada - A break up of the 2 networks
The second is not likely to happen. Rogers/Bell/Telus are some of the wealthiest companies in the world, and are at the top in terms of deep pockets in Canada. They will essentially fight the government on any sort of attempt to break them up and handily win.
The government has a much better shot at changing the rules to allow foreign investment, but they don't want to do it. The reason that's the case is because the leadership of the telecom companies has many shared members with prominent politicians, and also outright owns all forms of media in the country except for CBC, Post Media, and Corus.
Post Media is a known right-leaning org and Corus is effectively owned by Shaw, so their sway in the country's politics is not as big as Bell Media and Rogers Media who both own much bigger networks.
That means that they can run a very effective smear campaign against any politician or part who dares to allow foreign competition to enter Canada.
Seems like a rather large impact. Maybe Russia really wants its gas turbine back? [0]
[0] - https://www.reuters.com/world/exclusive-ukraine-urges-canada...
From what I understand of their current network setup, if their base stations in a region had no connectivity then the starlink customers in that region would lose their connectivity too.
This is likely going to be able to be mitigated when Starlink inter-satellite communications is up and running and they can route packets in space.
Society is fragile
And some hospitals appear to have similar issues.
It's an act of war.
I'm wary we may not find out for a long time.
Debit cards were out, huge swaths of wireless and internet, this was a huge failure.
It is owned by a crazy family in a perpetual feud that could have inspired Succession. https://www.ctvnews.ca/entertainment/succession-actor-brian-...
https://www.wsj.com/articles/rogers-family-feud-creates-boar...
Rogers and Bell executives are more interested in their ego than their networks and use their oligopoly money to buy sports teams: https://www.theglobeandmail.com/sports/no-love-for-blue-jays...
Last time this happened, they blamed Ericsson, but let’s face facts, they aren’t competent: https://mobilesyrup.com/2021/04/20/nationwide-rogers-outage-...
Their cellular business used to be called Cantel 30 years ago and we called it Can’t Talk.
Their CIO is from Vodaphone. https://about.rogers.com/team_bios/jorge-fernandes/
They have no oversight because the CRTC executive are best friends with the big 3 telcos: https://www.iphoneincanada.ca/carriers/bell/crtc-chair-on-dr...
Interac (Debit) apparently has their primary and backup network services with Rogers. They need a new executive team ASAP.
"Hours before the Rogers outage began, an apparent cyberattack took down the U.S. Congress website. According to Fox News, the pro-Russia hacking group Killnet took credit for crashing the website around 10 p.m. Thursday evening."
I didn't see the first part, i.e. hackers taking credit for the US Congress attack.
So I don't see anyone actually taking credit for it.
That said - this major attack is happening 1) at the very same moment US Congress is attacked, and, 2) Canada is at this very moment deciding on whether or not to return gas turbines to Russia as part of a maintenance agreement. The Turbines are part of the 'banned goods' but Germany wants the gear to go back to Russia for a key pipeline. Canada announced today they will send the gear to Russia.
So it's hugely speculative, but I'm still inclined to believe this was a hack - it's just too hugely suspicious.
[1] https://theparadise.ng/canada-suffering-massive-nationwide-i...