Hacker News Reader

Comment by schmuckonwheels | Hacker News Reader

I see multiple posts here speculating on cyberattack—as opposed to "we pushed a bad configuration update which messed everything up irreparably"—you know, like it has been every other time before this.

E.g., Cloudflare, Meta (who in doing so also locked themselves out of the building), and didn't some bumbling major Canadian telco knock themselves offline for like a week not too long ago?

View on HN

One of these times they will be right and you will never hear the end of the time they were first to recognize the start of a cyberattack.

Your post reflects another online observation. With the rise of online sports books, this sort of predictive doomerism has flooded almost every team's online comment section. It no longer feel like fandom or community in the same way. Just lots of voices that will be glad to say, "I told you so," in the loss and crickets with the W. Wish there was some accountability mechanism for all the negative noise broadcasted into the channel.

In general, whats the expected cost of being loudly, obviously, publicly and obnoxiously wrong?

Keep in mind who the President is.

You know what they say about a stopped clock: it's wrong 1,438 times a day.

If it is a quartz watch, it might be 47,120,384 times per day.

better odds than the lottery!

Not if you're playing Florida Pick 4, which is supposed to be 1:10000, but for me seems 1:U+221E

When telcos get compromised again, the attackers should just take it down in service of this moronic fallacy.

https://www.wsj.com/politics/national-security/china-cyberat...

It works both ways, a lot of people also take the "nothing ever happens" position and it is true that most of the time "nothing ever happens", so by taking that position, they're right 99% of the time and sound smart

Yes, but the majority opinion is "nothing ever happens" by default for everything, not just tech outages. It's not about sounding smart, but getting ahead of grifters and clout chasers.

Yep I think it started as a reaction against "it's happening!" types and it is a lot less wrong, but it's still wrong.

The truth is things do occasionally happen and we should be prepared, even if most of the time they don't

That's not true though, sometimes it's inexplicitly DNS too.

My money is on expired certificate.

If so, successful test.

The great thing is that with all of the encrypted/signed DNS thingys, you now have not one, but multiple options to combine “invalid certificate” and DNS in one outage. You no longer need to choose!

it's always DNS

or BGP.

No no it’s gotta be “the database”

I worked at a major ISP and we had a similar situation where the North East went down and the RC was a fiber cut at a major node in Philly.

These network topologies are incredibly complex and edges you think wouldn't exist have ways of suddenly appearing when things go awry.

Yeah, the Canadian telco was Rogers. Total recovery took multiple days. From the Wikipedia writeup:

> In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network.

https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...

The major Canadian outage was Rogers in 2022: https://en.wikipedia.org/wiki/2022_Rogers_Communications_out...

I took my kids swimming that day and the pool couldn't take our money since the payment terminal was on the Rogers cellular network, so it was a free family swim.

Cyberattacks are a good scapegoat for any large incompetent non-tech company that is unable to admit a mistake. (tech companies are more open to admitting actual mistakes - and reluctant to disclose cyberattacks even if there actually was one - where as non-tech ones would rather allude to an attack than admit a mistake)

Cyberattack scenarios pretty much never make sense in case of complete outages; if you have the access required to cause such an outage it’s always more profitable to keep this access and use it for covert spying/targeted attacks or save it for later than to burn it by causing a massive, visible problem.

Verizon had issues routing calls to a provider I'm aware of yesterday, and had to make some sort of change today to fix it. I'm definitely thinking bad configuration update.

In a dead empire, sufficiently advanced rot is indistinguishable from malice.

Infrastructure in general seems worse than 20 years ago. Our talent for black-bagging dictators has never been stronger, though!

20 years ago much less of the infrastructure of everyday life depended on an always-on network connection. Smartphones in particular were a relatively niche product. I didn’t even have a cell phone (and not because I was too young), much less expect it to work all the time.

It's affecting every mobile carrier (ATT, TMO), not just Verizon

No, it's affecting Verizon and people on ATT & TMO trying to get in touch with the Verizon customers who are affected.

Note the tiny fraction of people reporting for ATT & TMO compared to Verizon on DownDetector: https://downdetector.com/

(You have to click the links to see actual magnitude because the graphs are scaled to show relative outage within a given service -- order of 150k for Verizon vs 1.5k for the others.)

Reply on news.ycombinator.com