Optus cell and internet down across Australia | Hacker News Reader

Optus cell and internet down across Australia

abc.net.au

34 pointsemmelaich22 comments

The outage apparently began 4am AEDT. It's now 11am. For 7 hours and counting - no network at all in large parts of the country including no mobile network at all (if not all - I haven't heard of anywhere where it is working fine).

The impact is huge. As others have mentioned it's impacting financial services networks and in Melbourne, it's impacting the Metro train network (they use Optus for their backup communications).

The very widespread nature makes me think "there's gotta be more than one failure here" e.g. can't just be a DNS issue. Follow up thought - "could this be a non-accident?" i.e. a deliberate outage on behalf of actors unknown?

Most likely related to BGP, they probably made an erroneous change leading to routing errors.

These things are hard to fix and have been known to happen in big tech.

You can see a flood of BGP announcements the time it started: https://radar.cloudflare.com/as4804?dateRange=2d

The 4am start time kind of makes it sound like a scheduled change gone wrong. Maybe so wrong that they lost access to the control plane and needed to send engineers on-site to fix? I think if it had been malicious we would have heard about it by now..

Reports from a third party ISP is that Optus are flying technicians to at least eight sites across the country, and that it’s a problem with their BGP route reflectors. Facebook faced a similar problem last year IIRC.

“An Optus source, who did not wish to be named because they were not authorised to speak publicly, said a BGP prefix flood from a peer was likely causing the issues on the telco’s core network.”

“Our on-site technician is actively prioritising establishing a console connection [a physical cable connection],” the message to Optus’ partners early on Wednesday said. “Rest assured that said technician is also being provided additional technical support remotely.”

[1] https://www.smh.com.au/technology/what-caused-the-optus-outa...

Please educate us about BGP route reflectors. Any good YT explainers.

Down for over three hours so far, with no news about the fault. This is the largest internet outage in Australia's history.

Absolutely. Consumer and enterprise alike, EFTPOS, internet, landlines, mobiles, roaming, all down. Hospitals and train networks heavily impacted as well.

It baffles me how they can have one single point of failure for their entire national cellular + fibre voice & data network, and for it to take so long to fix.

If it’s BGP as is currently believed to be the case, it’s somewhat understandable. Same thing tripped up Facebook not too long ago.

You'd expect there would be redundancy in their core network which is why I'm leaning towards a DNS issue... It's always DNS after all.

How would that explain the voice/phone network being out? And you'd think packets would still be able to make it through the network even without DNS.

It’s BGP, not DNS.

Some sort of DNS issue? Maybe a botched change pushed out network wide? Surely if it was a change pushed out it would be rolled back by now?

Edit: On second thought, if they've got no network & no out-of-band - it may take a while... they may need to physically visit each site to re-install the backed-up version!

it's not DNS - there's no mobile network at all.

I think that’s how they respond to a major issue- they take down all the radio connections while they isolate the issue. Otherwise reconnect attempts will overwhelm the systems

Hm that makes sense.

BGP route reflectors seem to be the problem.

For Americans, this is comparable to Verizon going down. Optus is our second largest telco, with 10 million customers including many hospitals, trains, enterprises, etc… I myself am currently unable to make calls and use mobile internet.

From an interview with ABC (Australian public broadcaster/news network) with the Optus CEO ---

ABC: Can you tell us why the system's gone down?

Optus CEO: Unfortunately, I don't have more information to give at this stage. We have had issues since 4am. The team has tried a number of parts of restoration and so far we have not had the results that we have hoped for. And we're pursuing every avenue to get everybody back online as soon as possible.

ABC: Do they know what happened though? Who's there the team that you've got working on it?

Optus CEO: Our team is still pursuing every possible avenue. We had a number of hypotheses and each one so far that we've tested and put in place new actions for has not resolved the fundamental issue. So we're still working on it. And when we have a identified root cause and a time for restoration, we'll be updating everybody as soon as we can.

---

Pretty wild to be down for over 7 hours no with no sign of a fix yet. Note they have also said they don't believe it's a 'cyber attack' (Optus had an enormous data breach last year with 9.7 million customers data leaked: https://en.wikipedia.org/wiki/2022_Optus_data_breach )

Anecdotally, ATMs are down as well as EV chargers.

There are more comments to the similar story here: https://news.ycombinator.com/item?id=38184761

Reply on news.ycombinator.com