Hacker News Reader

Comment by yoelo | Hacker News Reader

Seriously? Is that how it works?

No. A network like Facebook's is vast and complicated and managed by higher-level configuration systems, not people emailing patches around.

If this issue is even to do with BGP it's much more likely the root of the problem is somewhere in this configuration system and that fixing it is compounded by some other issues that nobody foresaw. Huge events like this are always a perfect storm of several factors, any one or two of which would be a total noop alone.

The Swiss cheese model of accidents. Occasionally the holes all align.

https://en.wikipedia.org/wiki/Swiss_cheese_model

The fun part of BGP is they apparently make a lot of use of it within their network, not just advertising routes externally.

https://engineering.fb.com/2021/05/13/data-center-engineerin...

(and yes, fb.com resolves)

No, the backbone of the internet is not maintained with patches sent in emails.

https://lkml.org/

You are very wrong about that https://lkml.org/

Clearly you and the person you replied to are talking about very different things.

You are very wrong about that ;) https://lkml.org/

I think the sub-comment is confusing the linux kernel with BGP.

In a way, the Linux kernel does power the "backbones of the internet".

There are a hell of a lot of non-linux OS's running on core routers, but yes, in a way. However BGP isn't via email.

On the other hand, I and my office mate at the time negotiated the setup of a ridiculous number of BGP sessions over email, including sending configs. That was 20 years ago.

luckily not... would be absolutely terrible to have the backbone only on linux

Interoperability and a thriving ecosystem are necessities for resiliency.

Note that resiliency and efficiency are often working against each other.

I don't know. I doubt. It's just funny to think that you need email to fix BGP, but DNS is down because of BGP. You need DNS to send email which needs BGP. It's a kind of chicken and egg problem but at a massive scale this time.

Sheera Frenkel:

    Was just on phone with someone who works for FB who described employees unable to enter buildings this morning to begin to evaluate extent of outage because their badges weren’t working to access doors.

https://twitter.com/sheeraf/status/1445099150316503057

You'd think they'd have worked that into their DR plans for a complete P1 outage of the domain/DNS, but perhaps not, or at least they didn't add removal of BGP announcements to the mix.

Reply on news.ycombinator.com